On Sunday, 27 October 2024 21:39 UTC+7, SendTheSong was born. SendTheSong is a website where users can post anonymously & see messages for them with a twist of sharing your favorite song or just discovering new music.
The Genesis
It was just “another weekend side project” based on Propaganda Material that set by Razan 👀. At first, SendTheSong used this tech stack.
At first everything seems fine for a side project. Razan bought a domain (sendthesong.xyz) and registered on Cloudflare on Monday, 28 October 2024 13:51 UTC+7.
Also, Razan made an announcement on Monday, 28 October 2024 23:49 UTC+7 in Solopreneur Club ID Discord Server (with some censorship aligned with the propaganda material set by Razan).
And then, shared his creation to TikTok on Wednesday, 30 October 2024.
The Ascent
After the TikTok post, Razan saw a significant increase of unique users from SplitBee dashboard, up to 1K.
And then, still growing. Because of the limit of Free Plan in SplitBee, Razan migrated to PostHog. Also, SplitBee was down.
And still growing …
This was a screenshot from PostHog
Is this a DDoS?
:format(webp)/cdn.vox-cdn.com/uploads/chorus_asset/file/6438793/this-is-fine.jpg)
But after checking the TikTok post that was created by Razan before, it was really getting great user tractions.
Still constrained by the limit, Razan moved to Umami Analytics, hosted on Pikapods.
On Wednesday, 31 October 2024 08:56 UTC+7, SendTheSong was exceeding the Vercel Free Plan limit.
The Surge
Razan contacted me, Hanif, because of exceeding Free Plan limit issue. We teamed up for this project. For the second try, we used AWS with SST because the Free Tier limit in AWS seems bigger. At first, everything seemed to be working again, until Razan got some error when redeploying the website.
Also, Cloudfront usage in AWS was climbing alarmingly.
In a panic, Razan moved to Netlify. But the wave of users is not stopping. Free Plan in Netlify also got limited. Here are some screenshots of Usage in Netlify.
Meanwhile, we tried to ask Dax in SST Discord Server about the issue of failed deployment. We got a clear answer on Wednesday, 31 October 2024 19:52 UTC+7. It seems to be caused by deletion of something in AWS.
Now, AWS with SST problem has been solved. The next wave was incoming, impressively, if not terrifyingly.
On 1 November 2024, at 23:54 UTC+7, Razan tried to ask in Discord Server, named Solopreneur Club ID. This server is managed by M. Gilang Januar. Fortunately, we got some heartwarming responses, especially from Mas Rubi, the creator of Lokal.so.
We restructured the infrastructure again. The frontend part of SendTheSong still used S3+CloudFront combo, deployed with SST. The backend part, with guidance from Mas Rubi, Razan rewrote service with Go, deployed in AWS Lightsail.
Also, one of the most terrifying things in AWS. We exceeded the free tier limit.
The cost seems negligible right now. But it will be a disaster if not maintained correctly. So, we were moving to Cloudflare Pages for the front end.
Within only 4 hours after moving to Cloudflare Pages, we already reached the daily limit which is already exceeded more than seven times of the limit… (weirdly enough, Cloudflare Pages still running okay. A bug?)
For the backend side, Mas Rubi, gave some advice like migrating the database from MongoDB atlas to SQLite since there are a lot of connection errors.
After making the call with Mas Rubi, Razan migrated the database to SQLite and deployed it. But the CPU usage was 100% with VPS specifications of 2vCPU & 2GB RAM.
After checking the process, disk IO, and network bandwidth usage, the problem was found! The culprit was the usage of LIKE query. It takes a long time and sometimes crashes. So, we decided to optimize the query and removed the LIKE query.
if filter.Query != "" {
query = query.Where("recipient LIKE ?", "%"+filter.Query+"%")
}
recipient := strings.ToLower(post.Recipient)
if filter.Query != "" {
query = query.Where("recipient = ?", recipient)
}
To cover the server costs, since the website is free to use, we opened a donation page. We also had some issues regarding NextJS (that will be explained later).
Now, the page that we were afraid of most.
The AWS Budget Email …
While receiving the email, Razan already separated the frontend & backend. Razan removed the API Functions on Frontend and deployed the site on Cloudflare Pages. Also, Razan wrote the Backend using Go and deployed it on AWS Lightsail.
So, what happens after receiving the AWS Budget Email?
Razan immediately shut down all unused services on AWS. Razan also bought a VPS from BiznetGio and moved the Backend service from AWS Lightsail to BiznetGio VPS. At first, it was only 2 vCore 2GB RAM.
After a painful of migration and lots of setups, finally we can take some breath until….
until the new issues :)
We were facing new issues, the issue was on getting posts by query, we already had more than 3M posts at that time. Currently, the most popular features are posting & search/browsing. We had tried to implement the caching using Redis, but it didn’t work very well. Mas Rubi initiated to migrate all the data from SQLite to Badger.
After migrating to Badger, everything ran smoothly. We didn’t face any issues on the backend until this post was released.
But did you remember about the Cloudflare Workers daily limits above?
Well, it looks like the Cloudflare team already fixed that issue :).
We were facing another issue where the user can’t access post details of SendTheSong because this page was generated dynamically using Server-Side Rendering. We were using the NextJS with App Router. We were trying to find out why NextJS output did not build the frontend fully statically and deployed it to Cloudflare without reaching any limit.
The Turning Point
While looking for the answers and best solutions, Hanif found a remarkably interesting article!
After reading that article, we concluded the reason the frontend page is not exported statically. It was the NextJS App Router.
We tried to move from NextJS App Router to NextJS Pages Router and set up the configuration to make sure it exported fully statically.
import type { NextConfig } from "next";
const nextConfig: NextConfig = {
reactStrictMode: true,
output: 'export',
};
export default nextConfig;
Finally, it is fully static.
We deployed SendTheSong frontend to Cloudflare Pages and set up the _redirects to make sure the dynamic page keeps running on the Cloudflare Pages.
// _redirects file
/details/:id /details/[id] 200
After messing around finding the solution for days, we finally solved this last piece of puzzle.
The Epilogue
To conclude this laborious and backbreaking journey, we started with this stack.
And stay at this stack right now.
Frontend: NextJS; backend: Golang
Frontend hosted on Cloudflare Pages; backend hosted on VPS
Embedded DB with BadgerDB
Analytics with Cloudflare Web Analytics
Some key takeaways.
LIKE query is not optimal for search with exceedingly high users.
Do not be afraid of VPS.
Free Plan can be double-edged sword.
Fully static output can be done with NextJS Pages router.
It is possible to get dynamic pages with static export, but you need to set up redirection for hosting or web server.
Ask the right person, with the right question, at the right time.
A Haiku
An exploration
Sometimes overengineered
Let it be knowledge
This is not the end as we are still cooking new features for SendTheSong. Also, we would like to announce that SendTheSong will be maintained under 2SpacesDev, a developer community. And SendTheSong will still be free to use.
Additionally, we would like to express our highest gratitude to Mas M. Gilang Januar, Mas Rubi, and the many others whose names we cannot mention individually.
