Design Youtube - Scott

Design Youtube - Scott

https://www.youtube.com/watch?v=quJ7hwo5x8E

Requirements

Functional Requirements

  • Users should be able to upload videos.

  • Users should be able to share and view videos.

  • Users should be able to perform searches based on video titles.

  • Our services should be able to record stats of videos, e.g., likes/dislikes, total number of views, etc.

  • Users should be able to add and view comments on videos.

Non-Functional Requirements

  • The system should be highly reliable, any video uploaded should not be lost.

  • The system should be highly available. Consistency can take a hit (in the interest of availability); if a user doesn’t see a video for a while, it should be fine.

  • Users should have a **real time experience while watching videos **and should not feel any lag.

Bonus Part

  • most popular videos

  • Channels

  • Subscriptions

  • Trending videos

  • Tik Tok topics

  • Video recommendations

  • Real time 弹幕

Not include

  • Authentication / Authorization

CDN

Benefits of CDN

  1. Improving website load times - By distributing content closer to website visitors by using a nearby CDN server (among other optimizations), visitors experience faster page loading times.

  2. Reducing bandwidth costs.

  3. Increasing content availability and redundancy

  4. Improving website security

  5. TTL feature for certain use cases.

How to use CDN?

  1. Register with CDN

  2. Append resources with cdn url instead.

CDN Provide video streaming and subtitles/captions service.

Some Research

  • Twitch split each chunk as, 30 secs, 20KB, m3u8 format

  • Douyu CDN resource streaming 500KB/s flv

  • Youtube sample url for each chunk:

https://r1---sn-n4v7knlk.googlevideo.com/videoplayback?expire=1591856683&ei=y3nhXuCLFdXBkwbniKYw&ip=24.5.83.146&id=o-AHUeCZdeWM1OEkDzj1HyKxh5BEHkHITyr4H0fvBXrRuA&itag=247&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278&source=youtube&requiressl=yes&vprv=1&mime=video%2Fwebm&gir=yes&clen=74633294&dur=601.901&lmt=1591830739420755&fvip=1&keepalive=yes&beids=9466588&c=WEB&txp=6316222&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIgWqMCObQh4QU_Yi9fcUYe1MxqppBRZVeOkvD8KN1rNiECIQDxEKh3shZHw4Vi3g3vsiXgeng_YUJvLpvUMBP9odfe_A%3D%3D&alr=yes&cpn=V0OOFmt3RLVOJOt8&cver=2.20200609.04.01&redirect_counter=1&rm=sn-n4vl67e&fexp=9466588&cms_redirect=yes&ipbypass=yes&mh=pv&mip=2601:645:8101:1ea4:58e4:772f:34ad:760c&mm=31&mn=sn-n4v7knlk&ms=au&mt=1591835011&mv=m&mvi=0&pl=34&lsparams=ipbypass,mh,mip,mm,mn,ms,mv,mvi,pl&lsig=AG3C_xAwRQIhAIRB6XQ43hKSsrFl4FsXLn1WP94PtrS04HW1-0dmjWayAiBohaIotTy2bSOZYkFiGQeVqwylAU6th7Hbk__ZzFhT1Q%3D%3D&range=4664737-6134370&rn=19&rbuf=24672

Analysis using CDN

Stream video

Use CDN’s provided functionality to stream the video. No need to store any copy. Take cloudflare API as an example, it supports stream video, list streams, upload stream captions.

Sample returned result:

{
  "success": true,
  "errors": [],
  "messages": [],
  "result": [
    {
      "allowedOrigins": [
        "example.com"
      ],
      "created": "2014-01-02T02:20:00Z",
      "duration": 300,
      "input": {
        "height": 1080,
        "width": 1920
      },
      "maxDurationSeconds": 300,
      "meta": {},
      "modified": "2014-01-02T02:20:00Z",
      "uploadExpiry": "2014-01-02T02:20:00Z",
      "playback": {
        "hls": "https://videodelivery.net/ea95132c15732412d22c1476fa83f27a/manifest/video.m3u8",
        "dash": "https://videodelivery.net/ea95132c15732412d22c1476fa83f27a/manifest/video.mpd"
      },
      "preview": "https://watch.cloudflarestream.com/ea95132c15732412d22c1476fa83f27a",
      "readyToStream": true,
      "requireSignedURLs": true,
      "size": 4190963,
      "status": {
        "state": "inprogress",
        "pctComplete": 51
      },
      "thumbnail": "https://videodelivery.net/ea95132c15732412d22c1476fa83f27a/thumbnails/thumbnail.jpg",
      "uid": "ea95132c15732412d22c1476fa83f27a",
      "uploaded": "2014-01-02T02:20:00Z"
    }
  ],
  "total": "35586",
  "range": "1000"
}

Upload Video

  • TUS: Open protocol for reusable file uploads.

Traffic Estimation

Let’s assume we have 1.5 billion total users, 800 million of whom are daily active users. If, on average, a user views five videos per day then the total video-views per second would be:

800M∗5/86400sec=>46Kvideos/sec800M * 5 / 86400 sec => 46K videos/sec

Let’s assume our upload : view ratio is 1:200, i.e., for every video upload we have 200 videos viewed, giving us 230 videos uploaded per second.

46K/200=>230videos/sec46K / 200 => 230 videos/sec

Storage Estimates: Let’s assume that every minute 500 hours worth of videos are uploaded to Youtube. If on average, one minute of video needs 50MB of storage (videos need to be stored in multiple formats), the total storage needed for videos uploaded in a minute would be:

500hours∗60min∗50MB=>1500GB/min(25GB/sec)500 hours * 60 min * 50MB => 1500 GB/min (25 GB/sec)

These numbers are estimated by ignoring video compression and replication, which would change our estimates.

Bandwidth estimates: With 500 hours of video uploads per minute and assuming each video upload takes a bandwidth of 10 MB/min, we would be getting 300 GB of uploads every minute.

500hours∗60mins∗10MB=>300GB/min(5GB/sec)500 hours * 60 mins * 10MB => 300GB/min (5GB/sec)

Assuming an upload : view ratio of 1:200, we would need 1TB/s outgoing bandwidth. CDN could reduce 90% of traffic here.

High-level Design

Youtube is a complex system with lots of features and service. For separate of concerns purposes, the design of the youtube is separated into four major parts:

  • Video upload, processing, view

  • Metadata management (including user, # of views, likes, watching history, comment, channel, follow etc)

  • Streaming part (trending video, 弹幕，trending topic)

  • Video recommendation(personalized recommendation)

For separate of concerns principle, the following designs are broke into 4 parts as well:

Video Upload and Processing

Video is uploaded, chopped into 2 seconds clips, and stores the video clips in S3 alongside the Metadata. After the video is uploaded, compressed, chopped, stored and metadata processed & stored, return success to the user.

Then the video is set to ready for process, the process will generate a thumbnail(if user didn’t set), and index the video to make it searchable in the search engine.

The read will use CDN, if CDN is not buffered, read from Video metadata to identify the uuid of the video clips and then read from the S3.

API

Public API:

uploadVideo(api_dev_key, video_title, video_description, tags[], category_id, default_language, recording_details, video_contents)

  • api_dev_key (string): The API developer key of a registered account. This will be used to, among other things, throttle users based on their allocated quota.

  • video_contents (stream): Video to be uploaded.

searchVideo(search_query, user_location, maximum_videos_to_return, page_token)

getVideo(user_id, user_context, video_id, offset)

  • Videos are chopped to 2 seconds clips, and stored separately.

  • A typical short video of 5 mins = 150 clips. Medium size video 20 mins = 600 clips. Long video of 2 hours = 3600 clips.

Internal API:

GenerateThumbnail(video_id)

StoreThumbnail(video_id, image)

indexVideo(video_id)

Data Model

Video storage: SQL + S3

  • Next (few) uuid to reduce one additional call to metadata service

  • S3 PrimaryKey = { video id + clip id + resolution}

video_id
owner
timestamp
clip_id
offset
next_clip

uuid

bob

1235

uuid

125

next uuid

  • ElasticSearch index: skipped, each video is processed as a bag of words, similar to articles or webpages, use inverted index to index the video and BM25 to search the relevant videos. give terms and criteria, the elastic search will return the highly relevant video and then filter based on the other criterias like video length, upload time etc.

Video metadata SQL table

  • Next (few) uuid to reduce one additional call to metadata service

id
user
timestamp
title
description
total_length
clips_mapping
thumbnail
state
share_rule

uudi

bob

131546

Hello Word

system

151 sec

{{0:uuid0, s3url,...}...}

uuid

private/public group

  • State could be one of the following {NEW, UPLOADED, AVAILABLE, DELETED_PENDING, DELETED, STREAMING*},

    • NEW is when the user starts uploading a video.

    • When the video is uploaded, compressed, broken down and stored, the service changes the state into UPLOADED, async triggered the video post processing.

    • When the video is indexed in the search engine and passed all other checks(e.g. safety check), the service changed the state back to the AVAILABLE.

    • When the user deletes the video, the state will be changed to DELETED_PENDING, the search engine will un-index the video and then change it to DELETED.

    • STREAMING will be explained in the stream section.

  • Clips mappings could be normalized and stored in another table. This is particularly necessary if the video is streamed and will need to add new clips as they come. Note: frequent updates of the same row is really bad in system design.

Youtube Community

  • Popularity service in charge of the views, likes, dislikes of the video, likes and dislikes of the comments. The popularity service will periodically read all user activity logs and update the likes/views etc.

    • Popularity has massive read / write but ok with small chance of data loss, therefore use redis to store the data and async write back to DB.

  • All light weight(small data loss OK) activity go to the user activity logs （kafka）, that includes, view video clips, leave video pages, like a video etc.

    • The view of video will update view history and popularity separately.

    • One caveat is read after write could be inconsistent, either cached in the client side or read also from the last few seconds logs.

API

  • User CRUD

  • Follow / Unfollow

  • Channel CRUD

    • Playlists CRUD

    • Home CRUD

    • Features CRUD

  • Comment

    • AddComment(video_id, user, parent_comment_id, reply_to_user, comment)

      • timestamp will be added by the server

    • UpdateComment(video_id, user, comment_id, comment)

      • edited timestamp will be added by the server

Data Model

Comment table(DynamoDB)

  • partition key: videoId

  • sort key: commentId

  • local secondary Index on the parentCommentId.

videoId
commentId
timestamp
user
parentCommentId
comment
creation_timestamp
updated_timestamp
replyUser

uuid

uuid

564541

Bob

uuid

Very good video!

12345

254654

uuid

Popularity Table(Redis):

History Table(DyanmoDB)

  • view / like / dislike history

userId
timestamp
videoId
action_type
clip_uuid

uuid

12345

uuid

like/dislike/view

uuid

Streaming System

  • Features: Trending video / topics / streaming / live chat.

Trending Video / Topics could be done by adding a Flink, realtime stateful computation on top of data stream(user activities) to calculate.

Streaming requires prefill the metadata, thumbnail (if wanted), to make the stream searchable.

  1. The data is piped through the websocket to the server and then the server broadcasts the video clip bundled with the pointer to the next video clip.

  2. Users fresh and open the page will get the current clips and then subsequent reads will be based on the next video clip pointer, no need to request it from the server.

  3. If a connection is lost, reconnect to get the current clip url.

  4. Sending of live chat will be temp stored, bundled with video clip id.

  5. When the streamer streams the video, the live chat within a certain time threshold (< 5 seconds) will be displayed back asap.

  6. Viewers will watch the stream with a few seconds latency, therefore the live chat will be precisely displayed when the other user sends it.

Recommendation System

  • Every video is labeled with several tags, it could be done automatically or manually

    • System design, news, stock, yu qian, classical music, civ 6, food etc.

    • Some less popular page could be classified badly (bridge)

  • User preference profile could be built on top of user view/like/dislike of video ids. Then based on weight, an user profile could be built

    • Scott: {food:5.4, civ 6: 1.3, yu qian: 0.9, system design: 2.8}

  • A separate service could rank video by popularity from the popular service. Based on tags and the most popular video, a list of recommendations could be generated.

  • Collaborative filtering could be also used to do video recommendations.

  • The generated personal recommendations will be stored and saved in the database. When user login to the youtube page, the recommendations will render the page then.

PreviousDesigning Twitter SearchNextDesign live commenting

Last updated