An Erlang-based Hierarchical Distributed VoD System

Transcription

1 An Erlang-based Hierarchical Distributed VoD System Juan J. Sánchez, José L. Freire, Miguel Barreiro, Victor M. Gulías, Javier Mosquera LFCIA, University of Coruña An Erlang-based Hierarchical Distributed VoD System p.1/20

2 Mid 1999 Study with R, local cable provider. Interest in exploring technologies for a later VoD deployment. Market situation indicates VoD strategic in mid-term. Motivation Late 1999 Agreement LFCIA R to design and develop an experimental VoD system suitable for their fiber and cable network. Early 2000 EU grant to further support project R commitment to large scale deployment ? An Erlang-based Hierarchical Distributed VoD System p.2/20

3 Huge storage capacity High bandwidth (but relatively few requests) Reliable streaming (CBR, VBR...) Replacement for TV highly available VoD challenges Support for large and tiny configurations Adaptability to network topology Low cost (how much is people willing to pay?) An Erlang-based Hierarchical Distributed VoD System p.3/20

4 Existing VoD systems Most VoD servers designed for either low-bandwidth on the internet or high-bandwidth corporate LAN Cable operator customers: very heterogeneous, links from few hundred kbit/s to many Gbit/sec (56kbps dialups, kbps cable modems, fiber...) Most VoD servers designed upon a large, powerful, expensive storage server Cable network: need for distributed service due to network topology An Erlang-based Hierarchical Distributed VoD System p.4/20

5 Existing VoD systems (2) Usually tied to one or few individual video formats, codecs and transports Cable users: heterogeneous users (set-top users with existing MPEG decoders, next-generation set-tops, many different computer platforms, kiosks). An Erlang-based Hierarchical Distributed VoD System p.5/20

6 3-level hierarchy, specialized levels: Massive storage Distributed cache, with replication support Buffering, streaming, protocol adaptation Initial design proposal Specialized distributed hierarchical storage system. Focus on moving high volume bulk data around with specialized protocols, not on video. As flexible as possible An Erlang-based Hierarchical Distributed VoD System p.6/20

7 Physical structure Tape loaders, jukeboxes... Tertiary level (massive storage) Cluster nodes with local storage Secondary level (large scale cache) Cluster heads Primary level (buffering and protocol adaptation) Output streams An Erlang-based Hierarchical Distributed VoD System p.7/20

8 Initial design proposal(2) (replica) Cache in primary network node Streamer in final node End users PRIMARY RING Cache in primary network node Streamer in final node (main) Streamer in final node End users Cache Streaming An Erlang-based Hierarchical Distributed VoD System p.8/20

9 Choosing a development platform: Functional background, previous experience with distributed functional languages Java experience (thus we tend to avoid it) Portability a must Small group of developers Iterative development cycle Brewing VoDKA Need performance, but robustness more important Erlang open-sourced Finally: Erlang/OTP (...mostly) An Erlang-based Hierarchical Distributed VoD System p.9/20

10 Key technologies Erlang/OTP: short development time, few errors, thus high productivity. Developer training is critical. Restart coding from scratch? No panic. Linux: high performance, great portability, robustness, source availability, homogeneous license, fast evolution. Development with cheap hardware, deployment over anything. Standard protocols: adhere to publicly documented standards. An Erlang-based Hierarchical Distributed VoD System p.10/20

11 First lessons learnt OTP: high productivity, but steep initial learning curve Video formats, codecs, transport protocols nightmare: they change and evolve very rapidly, so don t become dependant on one. Ship soon, improve later. Too much time spent on insignificant details (many dumped afterwards). Exposure to real environment will improve product far more (but be responsive!) Need for further flexibility. Possibility of small mostly independent systems. Decentralized storage: multiple content providers An Erlang-based Hierarchical Distributed VoD System p.11/20

12 Design refinement Complete overhaul: attempt to generalize and better suit actual users needs. Arbitrary number of levels: small specialized modules (pattern: chain of responsability). VoD server cluster as storage subsystem for another server. I/O abstraction. Easy to add new transport protocols. Reflective servers (introspection). Separation of management application and monitoring from the VoD kernel. An Erlang-based Hierarchical Distributed VoD System p.12/20

13 VoDKA structure VODKA Monitor Monitor VODKA_slave VODKA_slave VODKA_slave Monitor Stream Group Streaming Sched Cache Sched n cache levels Sched Group RTP H.263 HTTP Frontend HTTP Frontend HTTP Streamer Cache Group Driver (File) Driver (TAPE) Group Cache Driver (File) Cache Driver (File) Driver (HTTP) An Erlang-based Hierarchical Distributed VoD System p.13/20

14 Physical topology example High bandwidth corporate user Group Streamer in secondary node (replica) Cache in primary network node Cache in secondary network node Streamer in final node End users PRIMARY RING Cache in primary network node Cache in secondary network node Streamer in final node (main) Streamer in final node End users L2 Cache L1 Cache Streaming An Erlang-based Hierarchical Distributed VoD System p.14/20

15 Second release changes Topology flexibility goal achieved Different video object transport protocols implemented Run-time selection of transport protocols VoDKA explorer Web end-user interface (movie catalog, etc) More streaming protocols An Erlang-based Hierarchical Distributed VoD System p.15/20

16 Ongoing and future work Job scheduling, job scheduling, job scheduling... (resource allocation) Work on mathematical workload models Plenty of room for optimization Make it more robust when something goes seriously wrong (hardware failing badly) Complex cache behaviour Reimplementing from scratch :-) An Erlang-based Hierarchical Distributed VoD System p.16/20

17 Erlang : Final lessons learnt... to date The module system is a bit simplistic Behaviours should be extensible without recompiling erlang itself Dynamic type system is a bit frustrating (vs. Caml, Haskell type systems) Trained developers productivity is great Surprisingly good performance. I/O subsystem now 100% Erlang. An Erlang-based Hierarchical Distributed VoD System p.17/20

18 Final lessons learnt... to date (2) OTP : Quite extensive (but far from perl or java libraries) Quite robust (far more than average perl or java) Somewhat steep learning curve. Doing individual tasks is well documented, gluing them together is not so well. Coherent and rather homogeneous. An Erlang-based Hierarchical Distributed VoD System p.18/20

19 Linux : Final lessons learnt... to date (3) Linux is cheap, Solaris/AIX/IRIX/whatever is more robust but expensive : myth. Linux is robust and not too expensive (but not cheap) Linux is extremely portable (Alpha, SPARC, HP-PA, x86, ia64, SH-3, ARM, m68k, s/390) but x86 is more on the safe side. Understanding the internals and tweaking things does make a difference Very good experience overall. An Erlang-based Hierarchical Distributed VoD System p.19/20

20 An Erlang-based Hierarchical Distributed VoD System Juan J. Sánchez, José L. Freire, Miguel Barreiro, Victor M. Gulías, Javier Mosquera LFCIA, University of Coruña An Erlang-based Hierarchical Distributed VoD System p.20/20