Newsletter: https://hw.glich.co/subscribe?utm_source=bluesky-profile
Challenges: global regulatory compliance, real-time fraud detection, massive traffic spikes (Black Friday!), and consistency across distributed databases.
Challenges: global regulatory compliance, real-time fraud detection, massive traffic spikes (Black Friday!), and consistency across distributed databases.
How? Google Search works in 4 steps:
- Crawling: Googlebots explore the web, follow links, respect robots.txt
- Indexing: Pages analyzed & catalogued in a massive searchable index
How? Google Search works in 4 steps:
- Crawling: Googlebots explore the web, follow links, respect robots.txt
- Indexing: Pages analyzed & catalogued in a massive searchable index
Unified data → Payments DB, Capital DB, and custom lists synced asynchronously into Elasticsearch
Cross-index matching → in-memory joins handle fuzzy/multi-source comparisons
Unified data → Payments DB, Capital DB, and custom lists synced asynchronously into Elasticsearch
Cross-index matching → in-memory joins handle fuzzy/multi-source comparisons
- Client apps request post metadata via the API layer
- API queries Aurora Postgres (partitioned by post_id)
- PgBouncer manages DB connections efficiently
- Metadata stored in JSONB fields → all info in 1 query, no joins
- Kafka CDC ensures writes consistent during migration
- Client apps request post metadata via the API layer
- API queries Aurora Postgres (partitioned by post_id)
- PgBouncer manages DB connections efficiently
- Metadata stored in JSONB fields → all info in 1 query, no joins
- Kafka CDC ensures writes consistent during migration
India's UPI changed everything: ₹500 to anyone in 3 seconds using just a phone number. Here's how it works under the hood:
Not a wallet: It's a real-time bank-to-bank infrastructure (345+ banks, 24/7)
India's UPI changed everything: ₹500 to anyone in 3 seconds using just a phone number. Here's how it works under the hood:
Not a wallet: It's a real-time bank-to-bank infrastructure (345+ banks, 24/7)
- Built on Django for rapid iteration, iOS-only launch
- AWS migration: EC2, S3, CloudFront, Elastic Load Balancing
- Data strategy: PostgreSQL + NoSQL, caching with Redis/Memcached, sharding & replication
- Built on Django for rapid iteration, iOS-only launch
- AWS migration: EC2, S3, CloudFront, Elastic Load Balancing
- Data strategy: PostgreSQL + NoSQL, caching with Redis/Memcached, sharding & replication
Airbnb’s User Signals Platform captures millions of user actions per second: searches, clicks, wishlist saves, and bookings.
Airbnb’s User Signals Platform captures millions of user actions per second: searches, clicks, wishlist saves, and bookings.
Asymmetric Encryption → Public key locks, private key unlocks (secure, powers WhatsApp end-to-end encryption & HTTPS)
Read full article: hw.glich.co/p/what-is-e...
Asymmetric Encryption → Public key locks, private key unlocks (secure, powers WhatsApp end-to-end encryption & HTTPS)
Read full article: hw.glich.co/p/what-is-e...
The secret: Encryption. It scrambles your message into unreadable ciphertext while it travels the internet.
Only the person with the right key can unlock it back into readable plaintext.
The secret: Encryption. It scrambles your message into unreadable ciphertext while it travels the internet.
Only the person with the right key can unlock it back into readable plaintext.
Also programmers:
Also programmers:
- Hot-standby Task Managers ready to take over if the primary fails
- Config-based workflows let teams define transforms without deep Flink knowledge
- Batch processing catches anything real-time missed.
No single point of failure
- Hot-standby Task Managers ready to take over if the primary fails
- Config-based workflows let teams define transforms without deep Flink knowledge
- Batch processing catches anything real-time missed.
No single point of failure
Session windows track short-term behavior patterns. Sliding windows provide continuous insights.
Dynamic categorization at scale
Session windows track short-term behavior patterns. Sliding windows provide continuous insights.
Dynamic categorization at scale
→ User browses listings
→ Events hit Kafka
→ Flink transforms raw events into meaningful signals
→ Stored in a key-value database with append-only writes
→ Other Airbnb services query this data for recommendations.
All under 1 second!
→ User browses listings
→ Events hit Kafka
→ Flink transforms raw events into meaningful signals
→ Stored in a key-value database with append-only writes
→ Other Airbnb services query this data for recommendations.
All under 1 second!
Speed Layer (Apache Flink) processes events instantly as they happen.
Batch Layer handles historical data for consistency checks, Serving Layer stores processed data for lightning-fast queries.
Best of both: speed + accuracy
Speed Layer (Apache Flink) processes events instantly as they happen.
Batch Layer handles historical data for consistency checks, Serving Layer stores processed data for lightning-fast queries.
Best of both: speed + accuracy
• Crawling (discovering pages)
• Indexing (organizing info)
• Ranking (ordering results)
• Serving (showing you results).
Each step is a masterpiece of engineering on a massive scale.
• Crawling (discovering pages)
• Indexing (organizing info)
• Ranking (ordering results)
• Serving (showing you results).
Each step is a masterpiece of engineering on a massive scale.
→ You type a URL
→ Browser does DNS lookup (finds IP address)
→ Establishes TCP connection
→ Sends HTTP request
→ Gets response.
This happens billions of times daily.
→ You type a URL
→ Browser does DNS lookup (finds IP address)
→ Establishes TCP connection
→ Sends HTTP request
→ Gets response.
This happens billions of times daily.
Service-Level: If video catalog is struggling, throttle only those requests
Global-Level: If Zuul itself is overwhelmed, shed traffic across ALL services
This prevents total Netflix outages
Service-Level: If video catalog is struggling, throttle only those requests
Global-Level: If Zuul itself is overwhelmed, shed traffic across ALL services
This prevents total Netflix outages
When systems get stressed, Zuul monitors:
• CPU usage
• Concurrent requests
• Connection counts
If thresholds are hit, it progressively drops low-priority requests using a cubic function for smooth scaling
When systems get stressed, Zuul monitors:
• CPU usage
• Concurrent requests
• Connection counts
If thresholds are hit, it progressively drops low-priority requests using a cubic function for smooth scaling