Adding subscribers to Substack

Or dealing with sunk costs and escalation of commitment...

This is a post from my Tech Journal section, in which I dive into technical topics, focusing on how I overcome specific challenges.
If you're not into the tech stuff, you can unsubscribe from Tech Journal updates from your account.

My build in public journey starts with technical challenges on stuff I wouldn't want to have to deal with…
I'm probably going down a rabbit hole here, but I hope I’ll be out of it soon.

The thing is, I have already invested some money in both Carrd & Substack, and I'd like the setup to work. Because I invested in it, right?
Right.

Check out the concept of sunk costs or escalation of commitment if you want to understand the dilemma I'm in 😅

Anyway.

So, I've decided to use Carrd.co to quickly build the landing page for my projects, and Substack to power my newsletter.

The issue is: since Substack has no public API, there’s no way to register subscribers on Substack from Carrd forms.

SubstackAPI.com

While searching for a potential solution, I stumbled upon SubstackAPI.com.

I've tried using their solution, but it ended up not working after a short while and I'm not sure why. The author's account is not very active, so I'm going to assume their service is broken…

Which leaves me no choice but to figure it out myself.

All by myself

Substack does have an API to register new subscribers. It's just not public.
So I'm taking my chances trying to use it myself, and I'm guessing that's just what substackapi.com did initially.

How Substack does it

Using the “splash screen” subscription form for my publication allowed me to get the details of the API call they're making to their servers to add a new subscriber. Here it is as a cURL command:

The important part is of course the JSON data at the end of the command, but interestingly Substack will not process the subscriber when I remove too much of the headers.

With trial and error, I got the request stripped down to:

Surprisingly, the user-agent header is required for the email to be sent. I guess it's one of their heuristics to prevent server-to-server signups 🤔

The origin header is also definitely used to check where the request is coming from and reject unrecognized sources. Since browsers won't let you override this header in JavaScript fetch requests, it's a good filter for blocking rogue forms from external sources.
However it does mean that I'll have to send the request from a server for things to work.

Making the call

Using Carrd’s custom form handler

It's possible, using Carrd, to directly address a remote server with a custom JSON payload. What's nice is, they make the API call from their own servers.
So I wouldn't have anything to code or host if it works!

I have dropped the accept and content-type headers as I suppose they'll be added by Carrd:

It took me 2 minutes to set this up on my Carrd form. Will it work?

👉 Hit save
👉 Open landing page
👉 Enter email & click “get updates” …
🥁
🦗🦗🦗

The call seems to have been successful, but I do not receive any subscription confirmation email from Substack.

Let's investigate!

As Carrd is sending the request from their servers, I cannot debug the API call from my browser. Using Beeceptor, I setup a mock endpoint to send the requests to, and inspect what data and headers Carrd is sending.

Here are the headers:

Both the referer and origin headers have been removed from the request. Damn.
Why would they do that I don’t know, but I'll have to use another method.

Plan B: all by myself, again

Plan B is quite simple: receive the form data from Carrd on a custom endpoint, and forward the subscription to Substack.

Since my techtrails.io domain is hosted at Cloudflare, I'm going to use their Workers feature to get this done quickly. I had never used Cloudflare as a development platform, and I'm surprised at how easy it is to get something up live!

All it takes is:

npm create cloudflare@latest

Then follow the instructions, log into my Cloudflare account and voilà! I have a running worker online.

Here’s the meaty part of what I deployed:

Deploying the new code is just a npm run deploy away, and I’m able to test the endpoint using the automated Swagger UI 👍

Aaaaand… it does not work.

I mean, the code works all right, it's pretty simple.
But Substack still completely ignores the subscriber request, even though it does respond with an HTTP 200 and returns some JSON payload response.

Using Beeceptor again as the target for this fetch request, I can see that everything is all right from a headers point of view.

Next, I just launched the node REPL from my machine and pasted the above fetch request into it.

It works! 🤯
With the exact. same. code.

So now I'm wondering whether they could be identifying that I'm calling them from a Cloudflare worker.
Another test from a VPS server led to the same result.
Could they be checking the originating IP address for well-known hosting providers?

Hard to know for sure, but it looks like the server-to-server calls are getting too complicated.

Plan C: it's all about origin

Call me stubborn, but I’ve got another trick up my sleeve.

The one thing I’m sure they won’t be filtering out is requests coming from a browser and originating from techtrails.io.

I've got one way to pull this out: iframe.

Cloudflare is an amazing tool. I can use it to re-route some traffic for my techtrails.io domain to a Worker, while letting the rest through to Substack.

I'm going to host a small HTML page on Cloudflare with some JS code that will send the subscription API call to Substack, and embed this into an iframe on my Carrd landing page.

With some cross-frame communication, I can trigger the subscription API call from my Carrd site but with Techtrails as the effective origin, and all should work.

Let's get to it!

On the Cloudflare side

I'm deploying a new Worker that will route everything to Substack normally, except a specific /substack-subscribers route that will serve my special page:

One npm run deploy later, and my domain is now serving this special page at techtrails.io/substack-subscribers.

But what does it contain?
Mainly JavaScript:

Here’s what the code does in a nutshell:

• Lines 14-64: Set up a listener for the message event and handle two different messages:

  • substack-subscribers:ack — simply to let the host page (the Carrd site) acknowledge that our iframe script is ready.

  • substack-subscribers:subscribe — to receive, from the host page, the email address that should be sent to Substack.

• Line 67: Post the substack-subscribers:init message to the host page. This will allow the host page to know which window & origin pair it should send its own messages to.

It will also post either :subscribed or :error messages to bubble up the Substack API call result to Carrd.

On Carrd side

On my Carrd landing page, I’m adding an Embed component to inject the following code into my page:

This code publishes a global Techtrails.subscribeEmail function (lines 40-50) that will be used by the Carrd custom Form to trigger the API call to Substack via the iframe.

It also handles the following messages:

• Lines 14-22: substack-subscribers:init – to auto-discover the Techtrails iframe to which messages should be sent.

• Lines 25-35: substack-subscribers:subscribed and :error — to handle subscription success or error by invoking Carrd’s callbacks. Those callbacks will in turn trigger the Carrd Form success or failure handlers, for instance to redirect to a success page after form submission.

Another Embed component allows me to inject the iframe that will load my techtrails.io/substack-subscribers page, responsible for communicating with Substack:

<iframe src="https://techtrails.io/substack-subscribers"></iframe>

Finally, I’m using a custom Carrd form set to run code upon submission:

The code calls my subscribeEmail function defined earlier, passing in the email to be subscribed and the success and failure callbacks to notify Carrd of submission completion.

Ok now. Save & publish.

The final test

Looks like my last idea was the one :)

Shortly after submitting the form on my landing page, I received the following email:

🥳

Out of the rabbit hole

Et voilà.
It was quite a journey to make this work!

Thank you for reading the whole thing.

I honestly wouldn’t have liked to rethink my whole landing page + newsletter setup, especially since I do appreciate Substack as a platform but the landing page design is quite limited.

However, I can’t help but feel that being a software engineer can sometimes be a curse in disguise: coding is a superpower I’m often tempted to use instead of looking for simpler solutions.

It’s definitely something I’ll have to be careful about on my journey to build myself a portfolio of side projects.

What about you? How often do you overdo stuff because you know you can do it?

Do let me know how you liked this deep dive into bypassing Substack’s (quite advanced) filters!