OK, the title might be a bit “click-baity” 😅

The truth is, Kamal is super cool to setup!
I have never deployed an app from scratch so quickly.

I did end up spending time into 4 rabbit holes though, so I wanted to keep notes about what happened and how to overcome the issues I faced.

You know, just in case.

Multi-arch build issues

The first issue I encountered was somewhat of a tricky one, although the fix was simple in the end.

It's always like this, isn't it? 😅

After I setup Kamal for my project, I ran kamal server bootstrap followed by kamal deploy, as instructed.

Colorful logs started flowing on my screen and I sat back, amazed by the simplicity of it all.

The Docker build process started. Along came the bundling step, and gem names came trickling down the terminal.

And then, it hung. For a few minutes.
I thought it must be doing something big, and it was late, so I went to bed.

The next morning, it was still stuck on the same thing:

 DEBUG [0729c69a] 	#29 668.9 Bundled gems are installed into `/usr/local/bundle`
 DEBUG [0729c69a] 	#29 668.9 Post-install message from rubyzip:
 DEBUG [0729c69a] 	#29 668.9 RubyZip 3.0 is coming!
 DEBUG [0729c69a] 	#29 668.9 **********************
 DEBUG [0729c69a] 	#29 668.9 
 DEBUG [0729c69a] 	#29 668.9 The public API of some Rubyzip classes has been modernized to use named
 DEBUG [0729c69a] 	#29 668.9 parameters for optional arguments. Please check your usage of the
 DEBUG [0729c69a] 	#29 668.9 following classes:
 DEBUG [0729c69a] 	#29 668.9   * `Zip::File`
 DEBUG [0729c69a] 	#29 668.9   * `Zip::Entry`
 DEBUG [0729c69a] 	#29 668.9   * `Zip::InputStream`
 DEBUG [0729c69a] 	#29 668.9   * `Zip::OutputStream`
 DEBUG [0729c69a] 	#29 668.9 
 DEBUG [0729c69a] 	#29 668.9 Please ensure that your Gemfiles and .gemspecs are suitably restrictive
 DEBUG [0729c69a] 	#29 668.9 to avoid an unexpected breakage when 3.0 is released (e.g. ~> 2.3.0).
 DEBUG [0729c69a] 	#29 668.9 See https://github.com/rubyzip/rubyzip for details. The Changelog also
 DEBUG [0729c69a] 	#29 668.9 lists other enhancements and bugfixes that have been implemented since
 DEBUG [0729c69a] 	#29 668.9 version 2.3.0.

A second run led to the same issue.

So I started searching, and found a few threads about the problem. The one that led me to a solution is Discussion #747 on Kamal’s repository: Deploy stucks on message about RubyZip 3.0 coming.
Someone in there hints at the issue coming from Kamal building a cross-platform Docker image by default, targeted for both AMD64 (Intel/x86-64 compatible processors) and ARM64 (ARM architectures such as Ampere processors).
Apparently building for ARM64 from an Intel CPU tends to fail and hang, at least on a Mac.

Since I am building on an Intel Mac but also for an x86-64 server, I don't need to build for ARM. So I disabled the ARM build, and everything went smoothly after that!

Everything?
Well, not exactly.

Using GHCR as Docker registry

As hinted above, Kamal uses Docker to build an app “artifact” and deploy it onto your servers. In the process, it needs to push the locally built image to a container registry, from which it can pull it again for deployment from your servers.

Since I'm using GitHub to host my code, I wanted to keep things simple and use their Container Registry feature as well, instead of Docker Hub.

It's pretty straightforward to setup Kamal for this:

The KAMAL_REGISTRY_PASSWORD comes from your .env file at the root of your project.
For things to work with GitHub Container Registry, you have to use a new Classic Personal Access Token (PAT) as your password.

I did this, but unfortunately it did not work immediately: as soon as Kamal tried to push the built image, it failed with an error:

Error: unexpected status from POST request to https://ghcr.io/v2/techtrails/real-emails/blobs/uploads/: 403 Forbidden

Kamal does manage to log into the registry as its first step, so the error was a bit unsettling: why could it log in, but not push an image to my repository registry?

Since I'm not pushing to my user GitHub (https://github.com/olance) but to my Techtrails organization (https://github.com/techtrails-io), I thought it might be an issue with my PAT and the rights it had to push to my org.

I spent a lot of time trying different tokens and tweaking settings on GitHub.
To no avail.

The 403 error really misled me into looking for token issues, but it was actually much simpler: as you might have noticed, I was pushing to techtrails/real-emails, whereas my GitHub org name is techtrails-io.

Yeah, techtrails was already taken :(

Kamal generates the push URL from the image setting in the deploy.yml file:

So I changed techtrails into techtrails-io, and it worked!

Subscribe now

TLS and CloudFlare

At this point, I had my app successfully deployed to real-emails.com.
It's there, you can check!

There were a few things, SSL/TLS-wise, that were not 100% satisfying.

TLS termination point & redirection loop

I'm using CloudFlare as my DNS server, taking advantage of their Proxy feature that serves my site through their own infrastructure, allowing them to cache content at the edge, detect DDoS attacks, etc.

They also provide automated TLS certificates out-of-the-box, which is great.
To make this simple for everyone, the way it works is that their servers act as a TLS termination point:

As illustrated above, a request coming from a client will be encrypted only between the browser and CloudFlare’s servers. CloudFlare relays the request to my servers in plain HTTP, without TLS encryption.

With a Rails application, what this causes when you load your website is an redirection loop, because Rails is configured to force SSL connections by default:

So when CloudFlare relays the browser’s request to Rails as an HTTP request, Rails responds with an HTTP Redirect from http://real-emails.com to https://real-emails.com.
This triggers a reloading of the HTTPS URL by the browser, which again is responded to with a redirect… and thus the loop begins.

My first fix to this issue was to turn config.force_ssl to false. Easy!

This, however, feels hacky and suboptimal. Plus it caused another issue that I forgot to note down.
And, I do want end-to-end encryption between the browser and my server.

So what I really want is …

CloudFlare “Full (strict)” encryption mode

In this mode, you get end-to-end encryption and a certificate provided by CloudFlare that allows them to certify they're talking to your origin server.

I think I'll write a specific tutorial to configure Kamal for this setup, but here’s the gist of it:

• Store CloudFlare certificate on your server

• Declare a Docker volume for the Traefik service to get access to this certificate and am accompanying TLS config file

• Configure a TLS entryPoint on Traefik and route it to your app’s service

Traefik is the reverse proxy Kamal is built upon to route requests to your application. In CloudFlare’s Full (strict or not) encryption mode, it's Traefik that becomes the TLS termination point, right before Rails serves your app.

So what you have to do, as outlined above, is just to provide it with the right TLS certificate and be done.
It’s even easier if you don't aim for strict mode (but I thought, let's go all the way!) since Traefik can provision certificates for you with Let's Encrypt.

Anyway. The simple yet terrible mistake I made here is that “Traefik” is simply the phonetics transcription of “traffic”, and some muscle memory at some point made me write “traefic” in the Docker volume name configuration 🤦‍♂️

Only after I left the broken site for a night and came back to it with fresh eyes and a thorough review of Traefik logs (which only indicated it hadn't found any certificate) did I understand my mistake.

Bonus: conflict with foreman

As an added bonus, I finally faced a seemingly unrelated issue when I got back to developing locally.

Since I'm using TailwindCSS, I have the cssbundling gem installed to rebuild my styles as I add new Tailwind classes to my templates.
This is handled by a separate process, so I'm using the provided bin/dev command instead of running rails server.

bin/dev is provided by the gem, and simply uses foreman to run both rails server and rails tailwindcss:watch in parallel.

What happened after I had completed my Kamal setup, is that:

• running rails server would work just fine

• running bin/dev would see the rails server process fail with an error: ActiveSupport::MessageEncryptor::InvalidMessage

This error generally occurs when Rails, upon booting, is not able to decrypt the credentials.yml.enc file that contains the secret_base token and other credentials you might have stored there.

It made no sense though, that it would work when run alone and fail when run through foreman.

To figure out what was going on, I edited Rails core files to place a debugger instruction right before the code that would throw the ActiveSupport::MessageEncryptor::InvalidMessage exception.

This allowed me to run rdbg (a command-line Ruby debugger) and attach to the process, which in turn let me inspect live variables content and see what the process was dealing with when crashing.

This is the code I dove into:

From decrypt_and_verify, in which the first catch_and_raise method was the one ultimately causing the crash, I could verify that the message argument contained the encrypted content of my development credentials.yml.enc file.
As I would expect in this context.

Later on, the read_message call leads to the decrypt method.
In there, I inspected the @secret instance variable, and found that it contained my production encryption key.

No surprise it couldn't correctly decrypt the file!

Now the question was: how did Rails, started in development mode, was picking up the production key instead of the development one?

After some thinking and digging, I inspected the content of ENV["RAILS_MASTER_KEY“], which is one way for Rails to get the decryption key.
It was indeed set to the production key.

That's when I realized what was going on:

• One one hand:
  When initializing Kamal, it creates a .env file into which you can provide environment variables for your deployment.
  If you recall from the beginning of this article, this is where the KAMAL_REGISTRY_PASSWORD value comes from.
  
  The other value that you need to put there, is the RAILS_MASTER_KEY encryption key, so that Kamal can set it on the deployed container for Rails to boot correctly.

• On the other hand:
  Foreman, used in the bin/dev script, will by default load environment variables for the spawned processes from any .env file it can find in its working directory.
  So it picks up the one from Kamal, and “corrupts” your local process with environment variables meant for production.

It's hard to tell whether Kamal or Foreman is at fault here, but once I understood the issue I quickly found a PR on GitHub that addressed the problem: Disable loading .env file when running foreman for development.

Simply tell foreman to load environment variables from /dev/null (or another arbitrary file if needed) in the bin/dev script:

The end.

My build in public journey starts with technical challenges on stuff I wouldn't want to have to deal with…
I'm probably going down a rabbit hole here, but I hope I’ll be out of it soon.

The thing is, I have already invested some money in both Carrd & Substack, and I'd like the setup to work. Because I invested in it, right?
Right.

Check out the concept of sunk costs or escalation of commitment if you want to understand the dilemma I'm in 😅

Anyway.

So, I've decided to use Carrd.co to quickly build the landing page for my projects, and Substack to power my newsletter.

The issue is: since Substack has no public API, there’s no way to register subscribers on Substack from Carrd forms.

SubstackAPI.com

While searching for a potential solution, I stumbled upon SubstackAPI.com.

I've tried using their solution, but it ended up not working after a short while and I'm not sure why. The author's account is not very active, so I'm going to assume their service is broken…

Which leaves me no choice but to figure it out myself.

All by myself

Substack does have an API to register new subscribers. It's just not public.
So I'm taking my chances trying to use it myself, and I'm guessing that's just what substackapi.com did initially.

How Substack does it

Using the “splash screen” subscription form for my publication allowed me to get the details of the API call they're making to their servers to add a new subscriber. Here it is as a cURL command:

The important part is of course the JSON data at the end of the command, but interestingly Substack will not process the subscriber when I remove too much of the headers.

With trial and error, I got the request stripped down to:

Surprisingly, the user-agent header is required for the email to be sent. I guess it's one of their heuristics to prevent server-to-server signups 🤔

The origin header is also definitely used to check where the request is coming from and reject unrecognized sources. Since browsers won't let you override this header in JavaScript fetch requests, it's a good filter for blocking rogue forms from external sources.
However it does mean that I'll have to send the request from a server for things to work.

Making the call

Using Carrd’s custom form handler

It's possible, using Carrd, to directly address a remote server with a custom JSON payload. What's nice is, they make the API call from their own servers.
So I wouldn't have anything to code or host if it works!

I have dropped the accept and content-type headers as I suppose they'll be added by Carrd:

It took me 2 minutes to set this up on my Carrd form. Will it work?

👉 Hit save
👉 Open landing page
👉 Enter email & click “get updates” …
🥁
🦗🦗🦗

The call seems to have been successful, but I do not receive any subscription confirmation email from Substack.

Let's investigate!

As Carrd is sending the request from their servers, I cannot debug the API call from my browser. Using Beeceptor, I setup a mock endpoint to send the requests to, and inspect what data and headers Carrd is sending.

Here are the headers:

Both the referer and origin headers have been removed from the request. Damn.
Why would they do that I don’t know, but I'll have to use another method.

Plan B: all by myself, again

Plan B is quite simple: receive the form data from Carrd on a custom endpoint, and forward the subscription to Substack.

Since my techtrails.io domain is hosted at Cloudflare, I'm going to use their Workers feature to get this done quickly. I had never used Cloudflare as a development platform, and I'm surprised at how easy it is to get something up live!

All it takes is:

npm create cloudflare@latest

Then follow the instructions, log into my Cloudflare account and voilà! I have a running worker online.

Here’s the meaty part of what I deployed:

Deploying the new code is just a npm run deploy away, and I’m able to test the endpoint using the automated Swagger UI 👍

Aaaaand… it does not work.

I mean, the code works all right, it's pretty simple.
But Substack still completely ignores the subscriber request, even though it does respond with an HTTP 200 and returns some JSON payload response.

Using Beeceptor again as the target for this fetch request, I can see that everything is all right from a headers point of view.

Next, I just launched the node REPL from my machine and pasted the above fetch request into it.

It works! 🤯
With the exact. same. code.

So now I'm wondering whether they could be identifying that I'm calling them from a Cloudflare worker.
Another test from a VPS server led to the same result.
Could they be checking the originating IP address for well-known hosting providers?

Hard to know for sure, but it looks like the server-to-server calls are getting too complicated.

Plan C: it's all about origin

Call me stubborn, but I’ve got another trick up my sleeve.

The one thing I’m sure they won’t be filtering out is requests coming from a browser and originating from techtrails.io.

I've got one way to pull this out: iframe.

Cloudflare is an amazing tool. I can use it to re-route some traffic for my techtrails.io domain to a Worker, while letting the rest through to Substack.

I'm going to host a small HTML page on Cloudflare with some JS code that will send the subscription API call to Substack, and embed this into an iframe on my Carrd landing page.

With some cross-frame communication, I can trigger the subscription API call from my Carrd site but with Techtrails as the effective origin, and all should work.

Let's get to it!

On the Cloudflare side

I'm deploying a new Worker that will route everything to Substack normally, except a specific /substack-subscribers route that will serve my special page:

One npm run deploy later, and my domain is now serving this special page at techtrails.io/substack-subscribers.

But what does it contain?
Mainly JavaScript:

Here’s what the code does in a nutshell:

• Lines 14-64: Set up a listener for the message event and handle two different messages:

  • substack-subscribers:ack — simply to let the host page (the Carrd site) acknowledge that our iframe script is ready.

  • substack-subscribers:subscribe — to receive, from the host page, the email address that should be sent to Substack.

• Line 67: Post the substack-subscribers:init message to the host page. This will allow the host page to know which window & origin pair it should send its own messages to.

It will also post either :subscribed or :error messages to bubble up the Substack API call result to Carrd.

On Carrd side

On my Carrd landing page, I’m adding an Embed component to inject the following code into my page:

This code publishes a global Techtrails.subscribeEmail function (lines 40-50) that will be used by the Carrd custom Form to trigger the API call to Substack via the iframe.

It also handles the following messages:

• Lines 14-22: substack-subscribers:init – to auto-discover the Techtrails iframe to which messages should be sent.

• Lines 25-35: substack-subscribers:subscribed and :error — to handle subscription success or error by invoking Carrd’s callbacks. Those callbacks will in turn trigger the Carrd Form success or failure handlers, for instance to redirect to a success page after form submission.

Another Embed component allows me to inject the iframe that will load my techtrails.io/substack-subscribers page, responsible for communicating with Substack:

<iframe src="https://techtrails.io/substack-subscribers"></iframe>

Finally, I’m using a custom Carrd form set to run code upon submission:

The code calls my subscribeEmail function defined earlier, passing in the email to be subscribed and the success and failure callbacks to notify Carrd of submission completion.

Ok now. Save & publish.

The final test

Looks like my last idea was the one :)

Shortly after submitting the form on my landing page, I received the following email:

🥳

Out of the rabbit hole

Et voilà.
It was quite a journey to make this work!

Thank you for reading the whole thing.

I honestly wouldn’t have liked to rethink my whole landing page + newsletter setup, especially since I do appreciate Substack as a platform but the landing page design is quite limited.

However, I can’t help but feel that being a software engineer can sometimes be a curse in disguise: coding is a superpower I’m often tempted to use instead of looking for simpler solutions.

It’s definitely something I’ll have to be careful about on my journey to build myself a portfolio of side projects.

What about you? How often do you overdo stuff because you know you can do it?

Do let me know how you liked this deep dive into bypassing Substack’s (quite advanced) filters!

I have been a Ruby on Rails fan for a very long time.

Unfortunately, my career track did not lead me to use it professionally, so it turns out I haven't touched Rails since its … 3rd version 🥲

Now that I am back to building personal projects, I've decided to pick up Rails again.

Meaning I have to install the entire stack from scratch, so here are the notes to achieve that1:

1. Install Ruby via rbenv

Looks like the landscape hasn't changed here and it's still RVM vs. rbenv!

I'm choosing rbenv for… reasons (mainly because it doesn’t mess with the environment/standard commands so much):

$ brew install rbenv 

Once installed, I can learn that the latest stable Ruby version is now 3.3.3:

$ rbenv install -l
3.1.6
3.2.4
3.3.3
jruby-9.4.7.0
mruby-3.3.0
picoruby-3.0.0
truffleruby-24.0.1
truffleruby+graalvm-24.0.1

Only latest stable releases for each Ruby implementation are shown.
Use `rbenv install --list-all' to show all local versions.

So let’s install it!

$ rbenv install 3.3.3

And make it the global Ruby version for my system:

$ rbenv global 3.3.3

For everything to work, don't forget to add the following to ~/.zshrc:

eval "$(rbenv init - zsh)"

2. Install Rails

As easy as it gets:

$ gem install rails

And voilà ! Ready to go 👍