Building Scalable Web Apps with Google App Engine Brett Slatkin June 14, 2008
Agenda Using the Python runtime effectively Numbers everyone should know Tools for storing and scaling large data sets Example: Distributed counters Example: A blog
Prevent repeated, wasteful work
Prevent repeated, wasteful work Loading Python modules on every request can be slow Reuse main() to addresses this: def main(): wsgiref.handlers.cgihandler().run(my_app) if name == " main ": main() Lazy-load big modules to reduce the "warm-up" cost def my_expensive_operation(): import big_module big_module.do_work() Take advantage of "preloaded" modules
Prevent repeated, wasteful work 2 Avoid large result sets In-memory sorting and filtering can be slow Make the Datastore work for you Avoid repeated queries Landing pages that use the same query for everyone Incoherent caching Use memcache for a consistent view: results = memcache.get('main_results') if results is None: results = db.gqlquery('...').fetch(10) memcache.add('main_results', results, 60)
Numbers everyone should know
Numbers everyone should know Writes are expensive! Datastore is transactional: writes require disk access Disk access means disk seeks Rule of thumb: 10ms for a disk seek Simple math: 1s / 10ms = 100 seeks/sec maximum Depends on: The size and shape of your data Doing work in batches (batch puts and gets)
Numbers everyone should know 2 Reads are cheap! Reads do not need to be transactional, just consistent Data is read from disk once, then it's easily cached All subsequent reads come straight from memory Rule of thumb: 250usec for 1MB of data from memory Simple math: 1s / 250usec = 4GB/sec maximum For a 1MB entity, that's 4000 fetches/sec
Tools for storing data
Tools for storing data: Entities Fundamental storage type in App Engine Schemaless Set of property name/value pairs Most properties indexed and efficient to query Other large properties not indexed (Blobs, Text) Think of it as an object store, not relational Kinds are like classes Entities are like object instances Relationship between Entities using Keys Reference properties One to many, many to many
Tools for storing data: Keys Key corresponds to the Bigtable row for an Entity Bigtable accessible as a distributed hashtable Get() by Key: Very fast! No scanning, just copying data Limitations: Only one ID or key_name per Entity Cannot change ID or key_name later 500 bytes
Tools for storing data: Transactions ACID transactions Atomicity, Consistency, Isolation, Durability No queries in transactions Transactional read and write with Get() and Put() Common practice Query, find what you need Transact with Get() and Put() How to provide a consistent view in queries?
Tools for storing data: Entity groups Closely related Entities can form an Entity group Stored logically/physically close to each other Define your transactionality RDBMS: Row and table locking Datastore: Transactions across a single Entity group "Locking" one Entity in a group locks them all Serialized writes to the whole group (in transactions) Not a traditional lock; writers attempt to complete in parallel
Tools for storing data: Entity groups 2 Hierarchical Each Entity may have a parent A "root" node defines an Entity group Hierarchy of child Entities can go many levels deep Watch out! Serialized writes for all children of the root Datastore scales wide Each Entity group has serialized writes No limit to the number of Entity groups to use in parallel Think of it as many independent hierarchies of data
Tools for storing data: Entity groups 3 Entity groups all transacting in parallel: Root Root Root Root Child Child Child Child Txn 1 Txn 2 Txn 3 Txn 4
Tools for storing data: Entity groups 4 Pitfalls Large Entity groups = high contention = failed transactions Not thinking about write throughput is bad Structure your data to match your usage patterns Good news Query across entity groups without serialized access! Consistent view across all entity groups No partial commits visible All Entities in a group are the latest committed version
Example: Counters
Counters Using Model.count() Bigtable doesn't know counts by design O(N); cannot be O(1); must scan every Entity row! Use an Entity with a count property: class Counter(db.Model): count = db.integerproperty() Frequent updates = high contention! Transactional writes are serialized and too slow Fundamental limitation of distributed systems
Counters: Before and after Single Sharded Counter Counter Counter Counter
Counters: Sharded Shard counters into multiple Entity groups Pick an Entity at random and update it transactionally Combine sharded Entities together on reads "Contention" reduced by 1/N Sharding factor can be changed with little difficulty
Counters: Models class CounterConfig(Model): name = StringProperty(required=True) num_shards = IntegerProperty(required=True, default=1) class Counter(Model): name = StringProperty(required=True) count = IntegerProperty(required=True, default=0)
Counters: Get the count def get_count(name): total = 0 for counter in Counter.gql( 'WHERE name = :1', name): total += counter.count return total
Counters: Increment the count def increment(name): config = CounterConfig.get_or_insert(name, name=name) def txn(): index = random.randint(0, config.num_shards - 1) shard_name = name + str(index) counter = Counter.get_by_key_name(shard_name) if counter is None: counter = Counter( key_name=shard_name, name=name) counter.count += 1 counter.put() db.run_in_transaction(txn)
Counters: Cache reads def get_count(name): total = memcache.get(name) if total is None: total = 0 for counter in Counter.gql( 'WHERE name = :1', name): total += counter.count memcache.add(name, str(total), 60) return total
Counters: Cache writes def increment(name): config = CounterConfig.get_or_insert(name, name=name) def txn(): index = random.randint(0, config.num_shards - 1) shard_name = name + str(index) counter = Counter.get_by_key_name(shard_name) if counter is None: counter = Counter(key_name=shard_name, name=name) counter.count += 1 counter.put() db.run_in_transaction(txn) memcache.incr(name)
Example: Building a Blog
Building a Blog Standard blog Multiple blog posts Each post has comments Efficient paging without using queries with offsets Remember, Bigtable doesn't know counts!
Building a Blog: Blog entries Blog entries with an index Having an index establishes a rigid ordering Index enables efficient paging This is a global counter, but it's okay Low write throughput of overall posts = no contention
Building a Blog: Models class GlobalIndex(db.Model): max_index = db.integerproperty(required=true, default=0) class BlogEntry(db.Model): index = db.integerproperty(required=true) title = db.stringproperty(required=true) body = db.textproperty(required=true)
Building a Blog: Posting an entry def post_entry(blogname, title, body): def txn(): blog_index = BlogIndex.get_by_key_name(blogname) if blog_index is None: blog_index = BlogIndex(key_name=blogname) new_index = blog_index.max_index blog_index.max_index += 1 blog_index.put() new_entry = BlogEntry( key_name=blogname + str(new_index), parent=blog_index, index=new_index, title=title, body=body) new_entry.put() db.run_in_transaction(txn)
Building a Blog: Posting an entry 2 Hierarchy of Entities: Blog Index Entry
Building a Blog: Getting one entry def get_entry(blogname, index): entry = BlogEntry.get_by_key_name( parent=key.from_path('blogindex', blogname), blogname + str(index)) return entry That's it! Super fast!
Building a Blog: Paging def get_entries(start_index): extra = None if start_index is None: entries = BlogEntry.gql( 'ORDER BY index DESC').fetch( POSTS_PER_PAGE + 1) else: start_index = int(start_index) entries = BlogEntry.gql( 'WHERE index <= :1 ORDER BY index DESC', start_index).fetch(posts_per_page + 1) if len(entries) > POSTS_PER_PAGE: extra = entries[-1] entries = entries[:posts_per_page] return entries, extra
Building a Blog: Comments High write-throughput Can't use a shared index Would like to order by post date Post dates aren't unique, so we can't use them to page: 2008-05-26 22:11:04.1000 Before 2008-05-26 22:11:04.1234 My post 2008-05-26 22:11:04.1234 This is another post 2008-05-26 22:11:04.1234 And one more post 2008-05-26 22:11:04.1234 The last post 2008-05-26 22:11:04.2000 After
Building a Blog: Comments High write-throughput Can't use a shared index Would like to order by post date Post dates aren't unique, so we can't use them to page: 2008-05-26 22:11:04.1000 Before 2008-05-26 22:11:04.1234 My post 2008-05-26 22:11:04.1234 This is another post 2008-05-26 22:11:04.1234 And one more post 2008-05-26 22:11:04.1234 The last post 2008-05-26 22:11:04.2000 After
Building a Blog: Composite properties Make our own composite string property: "post time user ID comment ID" Use a shared index for each user's comment ID Each index is in a separate Entity group Guaranteed a unique ordering, querying across entity groups: 2008-05-26 22:11:04.1000 brett 3 Before 2008-05-26 22:11:04.1234 jon 3 My post 2008-05-26 22:11:04.1234 jon 4 This is another post 2008-05-26 22:11:04.1234 ryan 4 And one more post 2008-05-26 22:11:04.1234 ryan 5 The last post 2008-05-26 22:11:04.2000 ryan 2 After
Building a Blog: Composite properties 2 High throughput because of parallelism User Index User Index User Index Comment Comment Comment
What to remember
What to remember Minimize Python runtime overhead Minimize waste Why Query when you can Get? Structure your data to match your load Optimize for low write contention Think about Entity groups Memcache is awesome-- use it!
Learn more code.google.com