Design Youtube - Scott
Design Youtube - Scott
https://www.youtube.com/watch?v=quJ7hwo5x8E
Requirements
Functional Requirements
Users should be able to upload videos.
Users should be able to share and view videos.
Users should be able to perform searches based on video titles.
Our services should be able to record stats of videos, e.g., likes/dislikes, total number of views, etc.
Users should be able to add and view comments on videos.
Non-Functional Requirements
The system should be highly reliable, any video uploaded should not be lost.
The system should be highly available. Consistency can take a hit (in the interest of availability); if a user doesn’t see a video for a while, it should be fine.
Users should have a **real time experience while watching videos **and should not feel any lag.
Bonus Part
most popular videos
Channels
Subscriptions
Trending videos
Tik Tok topics
Video recommendations
Real time 弹幕
Not include
Authentication / Authorization
CDN
Benefits of CDN
Improving website load times - By distributing content closer to website visitors by using a nearby CDN server (among other optimizations), visitors experience faster page loading times.
Reducing bandwidth costs.
Increasing content availability and redundancy
Improving website security
TTL feature for certain use cases.
How to use CDN?
Register with CDN
Append resources with cdn url instead.
CDN Provide video streaming and subtitles/captions service.
Some Research
Twitch split each chunk as, 30 secs, 20KB, m3u8 format
Douyu CDN resource streaming 500KB/s flv
Youtube sample url for each chunk:
Analysis using CDN
Stream video
Use CDN’s provided functionality to stream the video. No need to store any copy. Take cloudflare API as an example, it supports stream video, list streams, upload stream captions.
Sample returned result:
Upload Video
TUS: Open protocol for reusable file uploads.
Traffic Estimation
Let’s assume we have 1.5 billion total users, 800 million of whom are daily active users. If, on average, a user views five videos per day then the total video-views per second would be:
Let’s assume our upload : view ratio is 1:200, i.e., for every video upload we have 200 videos viewed, giving us 230 videos uploaded per second.
Storage Estimates: Let’s assume that every minute 500 hours worth of videos are uploaded to Youtube. If on average, one minute of video needs 50MB of storage (videos need to be stored in multiple formats), the total storage needed for videos uploaded in a minute would be:
These numbers are estimated by ignoring video compression and replication, which would change our estimates.
Bandwidth estimates: With 500 hours of video uploads per minute and assuming each video upload takes a bandwidth of 10 MB/min, we would be getting 300 GB of uploads every minute.
Assuming an upload : view ratio of 1:200, we would need 1TB/s outgoing bandwidth. CDN could reduce 90% of traffic here.
High-level Design
Youtube is a complex system with lots of features and service. For separate of concerns purposes, the design of the youtube is separated into four major parts:
Video upload, processing, view
Metadata management (including user, # of views, likes, watching history, comment, channel, follow etc)
Streaming part (trending video, 弹幕，trending topic)
Video recommendation(personalized recommendation)
For separate of concerns principle, the following designs are broke into 4 parts as well:
Video Upload and Processing
Video is uploaded, chopped into 2 seconds clips, and stores the video clips in S3 alongside the Metadata. After the video is uploaded, compressed, chopped, stored and metadata processed & stored, return success to the user.
Then the video is set to ready for process, the process will generate a thumbnail(if user didn’t set), and index the video to make it searchable in the search engine.
The read will use CDN, if CDN is not buffered, read from Video metadata to identify the uuid of the video clips and then read from the S3.
API
Public API:
uploadVideo(api_dev_key, video_title, video_description, tags[], category_id, default_language, recording_details, video_contents)
api_dev_key (string): The API developer key of a registered account. This will be used to, among other things, throttle users based on their allocated quota.
video_contents (stream): Video to be uploaded.
searchVideo(search_query, user_location, maximum_videos_to_return, page_token)
getVideo(user_id, user_context, video_id, offset)
Videos are chopped to 2 seconds clips, and stored separately.
A typical short video of 5 mins = 150 clips. Medium size video 20 mins = 600 clips. Long video of 2 hours = 3600 clips.
Internal API:
GenerateThumbnail(video_id)
StoreThumbnail(video_id, image)
indexVideo(video_id)
Data Model
Video storage: SQL + S3
Next (few) uuid to reduce one additional call to metadata service
S3 PrimaryKey = { video id + clip id + resolution}
| video_id | owner | timestamp | clip_id | offset | next_clip |
|---|---|---|---|---|---|
uuid | bob | 1235 | uuid | 125 | next uuid |
ElasticSearch index: skipped, each video is processed as a bag of words, similar to articles or webpages, use inverted index to index the video and BM25 to search the relevant videos. give terms and criteria, the elastic search will return the highly relevant video and then filter based on the other criterias like video length, upload time etc.
Video metadata SQL table
Next (few) uuid to reduce one additional call to metadata service
| id | user | timestamp | title | description | total_length | clips_mapping | thumbnail | state | share_rule |
|---|---|---|---|---|---|---|---|---|---|
uudi | bob | 131546 | Hello Word | system | 151 sec | {{0:uuid0, s3url,...}...} | uuid | private/public group |
State could be one of the following {
NEW,UPLOADED,AVAILABLE,DELETED_PENDING,DELETED,STREAMING*},NEWis when the user starts uploading a video.When the video is uploaded, compressed, broken down and stored, the service changes the state into
UPLOADED, async triggered the video post processing.When the video is indexed in the search engine and passed all other checks(e.g. safety check), the service changed the state back to the
AVAILABLE.When the user deletes the video, the state will be changed to
DELETED_PENDING, the search engine will un-index the video and then change it toDELETED.STREAMINGwill be explained in the stream section.
Clips mappings could be normalized and stored in another table. This is particularly necessary if the video is streamed and will need to add new clips as they come. Note: frequent updates of the same row is really bad in system design.
Youtube Community
Popularity service in charge of the views, likes, dislikes of the video, likes and dislikes of the comments. The popularity service will periodically read all user activity logs and update the likes/views etc.
Popularity has massive read / write but ok with small chance of data loss, therefore use redis to store the data and async write back to DB.
All light weight(small data loss OK) activity go to the user activity logs （kafka）, that includes, view video clips, leave video pages, like a video etc.
The view of video will update view history and popularity separately.
One caveat is read after write could be inconsistent, either cached in the client side or read also from the last few seconds logs.
API
User CRUD
Follow / Unfollow
Channel CRUD
Playlists CRUD
Home CRUD
Features CRUD
Comment
AddComment(video_id, user, parent_comment_id, reply_to_user, comment)timestamp will be added by the server
UpdateComment(video_id, user, comment_id, comment)edited timestamp will be added by the server
Data Model
Comment table(DynamoDB)
partition key: videoId
sort key: commentId
local secondary Index on the parentCommentId.
| videoId | commentId | timestamp | user | parentCommentId | comment | creation_timestamp | updated_timestamp | replyUser |
|---|---|---|---|---|---|---|---|---|
uuid | uuid | 564541 | Bob | uuid | Very good video! | 12345 | 254654 | uuid |
Popularity Table(Redis):
History Table(DyanmoDB)
view / like / dislike history
| userId | timestamp | videoId | action_type | clip_uuid |
|---|---|---|---|---|
uuid | 12345 | uuid | like/dislike/view | uuid |
Streaming System
Features: Trending video / topics / streaming / live chat.
Trending Video / Topics could be done by adding a Flink, realtime stateful computation on top of data stream(user activities) to calculate.
Streaming requires prefill the metadata, thumbnail (if wanted), to make the stream searchable.
The data is piped through the websocket to the server and then the server broadcasts the video clip bundled with the pointer to the next video clip.
Users fresh and open the page will get the current clips and then subsequent reads will be based on the next video clip pointer, no need to request it from the server.
If a connection is lost, reconnect to get the current clip url.
Sending of live chat will be temp stored, bundled with video clip id.
When the streamer streams the video, the live chat within a certain time threshold (< 5 seconds) will be displayed back asap.
Viewers will watch the stream with a few seconds latency, therefore the live chat will be precisely displayed when the other user sends it.
Recommendation System
Every video is labeled with several tags, it could be done automatically or manually
System design, news, stock, yu qian, classical music, civ 6, food etc.
Some less popular page could be classified badly (bridge)
User preference profile could be built on top of user view/like/dislike of video ids. Then based on weight, an user profile could be built
Scott: {food:5.4, civ 6: 1.3, yu qian: 0.9, system design: 2.8}
A separate service could rank video by popularity from the popular service. Based on tags and the most popular video, a list of recommendations could be generated.
Collaborative filtering could be also used to do video recommendations.
The generated personal recommendations will be stored and saved in the database. When user login to the youtube page, the recommendations will render the page then.
Last updated
