Design Ads Logging System

Design Ads Logging System

Requirements

Functional requirements

• Generic ads logging system

• Advertiser - provide ads - want users to convert

• Publisher - display ads

• Influence - ranking of the ads

• Log publisher page information, end user information

• calculate the count of different ads(click, display, etc.)

• Ad Event: display, click, conversion

Non-functional requirements

• Scalability

• Lower latency

• High availability

---

High-level Architecture

---

Detailed Design

Ads Publish Service

• user can publish new ads

• new ads request will send to Ads Service

• store ads metadata in the SQL database

Ads Table

adId(PK)	userID	description	created_time
123485	4541	"new ad"	456487

---

Ads Service

• rank ads

• other ads related operation

Search/Ads Ranking

• ads relevance check

• impression count

• CTR: Click-Through Rate

• CVR: Conversion Rate

---

Logging Service

• we can use lambda architecture to do handle calculation of how many counts for different ads

• Real-time processor is the speed layer - not that accurate count (calculate lastest 1 min, 5 min results)

• Batch Processor is the batch layer - accurate count (calculate recent 1 day, 1 month, 1 year results)

Message Request

{
    eventId: 1254, // specify a unique ad
    advertiserId: 15464,
    userId: 676467,
    eventType: "click",
    timestamp: 45613854,
}

Real-time Processor

• it contains multiple consumers to poll messages from the queue

• impression calculated by the stream processor; impression + 1 if it is the user

• sharding based on eventId or advertiserId

• each of the consumer can handle all events for the same eventId

• it is write-heavy, we can use Cassandra (append only) or ElastiSearch to store the results

• in the processor, we can also build in-memory hashtable

  • use LSM(log-structured merge-tree) as the data structure

Batch Processor

• log all the data

• map-reduce job for accurate data

• mainly for data correction and monitoring