*EDIT: So, I posted this article on HackerNews and the first thing people were kind enough to point out was that I could just use CGI. Good point!

I’ll take a look at that. However, I still had fun working on this side project, even if it ended up a small CGI clone in OpenResty! Thank you for reading!*

Everyone encounters a moment in which they’re held back by a limitation of a certain thing they use. Whether that’s your morning citrus juicer that doesn’t seem to catch all the pits in your wake-up juice.

We encounter all sorts of limitations in our day-to-day life. It doesn’t always have to be a frustration that causes you to solve these limitations. In my case – although unrelatable – it’s the limitation of not being able to run a bash script from an nginx location directive. For the purposes of this article, I’m assuming you know what all of those things are. If you don’t know what any of this means, don’t worry, but the rest of the article is probably not gonna make a lot of sense. (Not all the articles of this blog are for everyone I suppose.

When you use nginx, you usually use it as either a reverse proxy or a simple web hosting server. Most people don’t use it much outside of these purposes. Whether you run larger services, you are into devOps or you find it fun to poke around with stuff like this (I’m mostly the latter two), you can find yourself using a lot of other features too.

One of the features that I was looking for was a simple way to execute a bash script from one of the location directives of an nginx configuration. Usually I write a small service that responds through a reverse proxy. Although that was fun in the beginning, I’m starting to notice that my use cases are usually too small for writing a service. The truth is: I don’t enjoy making a side project out of these small problems. I found that writing a simple bash script is the best way to go about these smaller problems. I already do this on my computer by having a big ~/.bin folder with all sorts of scripts. The problem remains: how do you solve this on the internet when it comes to API’s and services? Well, you look for a way to execute a bash script through nginx.

Okay, so I have to be honest here. I was gonna write this part of the article showing how hard it was to find a solution. After making Neh and deciding to write the article, it seems that my first search query finds a good answer.

When I searched the first time, most of the answers that I found revolved around using things like “FastCGI with PHP”. I wasn’t gonna use PHP to call a bash script, that would be overdoing it probably. I also found a dead forum post on nginx.org with a poor citizen of the internet with the same question as I had.

Okay, so luckily for me, I ended up at a StackOverflow thread that had the answer to my itching question.

location /my-website {
  content_by_lua_block {
    os.execute("/bin/myShellScript.sh")
  } 
}

Ofcourse! Use the HTTPLuaModule from nginx! This is the answer to all my problems. I also wanted to know the output of the script, so my config ended up looking like this:

location /my-website {
  content_by_lua_block {
    local file = assert(io.popen("/bin/myShellScript.sh"))
    local result = assert(file:read('*all'))
    ngx.say(result)
    file:close()
  }
}

So yay! I can now execute my shell script straight from an nginx location directive!

But I couldn’t just leave it there. There was one particular application I wanted to use it for: GitHub webhooks. Particularly, I wanted it for the blog that you’re reading right now. This website runs on Hugo and needs to be compiled everytime there is a push. The code above is not flexible enough to support that use case.

So, I made sure that it was.

Neh is a small lua script that provides the perfect framework for your one-off scripts or even programs through nginx. It has the features of the above code and some extras like:

  • **Passing all of the request headers as Environment Variables. **So User-Agent becomes a readable USER_AGENT variable

  • **Send any data sent through the request as data through **stdin**! **Have a reliable cross-language/cross-program way to receive the data.

  • **Sending your **stdout** as a response, chunked instead of a big response buffer. **makes it easier to stream larger responses like files.

  • **Being able to write to fd #3 to write headers of the response. **You can easily manipulate the headers through a file descriptor by writing to it!

  • **A file descriptor (#4) for sending commands to Neh. **It makes Neh able to do actions on your behalf on the lower level.

The last feature is useful for things like ending the connection but continuing the script. All these features proved useful with GitHub webhooks, because they need a way to read the request headers, read the request JSON and send back a response immediately after.

Installing Neh can be done with this simple one-liner:

curl https://raw.githubusercontent.com/oap-bram/neh/master/install.sh | sh

Setting up Neh is easy! Just set a nginx variable in your location directive and let the content_by_lua_file point to Neh! The rest is done for you!

location /hooks/github-commit {
    set $execute_file /home/bram/blog/github-commit-hook.sh; # The file I want to execute
    content_by_lua_file /usr/lib/neh/neh.lua; # Execute the request and file with neh
}

Then I created a script, for the purpose of this blog. I’ve omitted the part that actually does the git pulling and the hugoing. This example just reads the body, and passes it through OpenSSL to make a HMAC hash. This is to verify the request is actually from GitHub.

#!/bin/bash
# Hook that is called when the github-commit hook is run

# Before we send any data I set the Content-Type to text/plain
# This is already done automatically by Neh, but I wanted to showcase the
# feature anyway 

echo "Content-Type: text/plain" >&3

# The secret you set up on GitHub
secret="a-secret-im-obviously-not-leaking-through-a-blog-post"

# Read the content of the body from stdin
body=$(cat <&0)

# Make a hash based on the body and digest into a sha1 HMAC as given by GitHub
hash="sha1=$(echo -n "$body" | openssl dgst -sha1 -hmac "$secret" | cut -d ' ' -f2)"

# Compare the hash with the request header generated by Neh
if [[ "$hash" != "$X_HUB_SIGNATURE" ]]; then
    # Fail the request if the signatures do not come across.
    echo "Hook verification failed"
    exit -1
fi

# Respond to the webhook with a message of success!
echo "Hook verification successful!"

# Immediately end the request afterward by writing to the command file
# descriptor #4
echo "END_REQUEST" >&4

# Write the rest of the output to /dev/null because we can't write to the
# response body anymore
exec >/dev/null
exec 2>/dev/null

# Actually do the rest of the work required to update the website

...

And that’s it! I made a relatively small script for a GitHub webhook like that! Nothing stops you from doing it in Ruby, Node.js, Python or even a compiled Go binary. You can execute any program with Neh and use the environment variables and stdin/stdout in your language/platform of choice!

Damn, have I learned a lot on this project. I thought setting this up would be somewhat trivial, but because of Lua’s close-to-C nature, I was quickly forced to go deep on most of the problems that I encountered.

Here are some problems that taught me a ton:

  • **Pipes aren’t supported by lua nor nginx lua out of the box. **Well, suppose I’ll just build my own support with luaposix. Sure learned how pipes work on a Unix level now.

  • **I need to run multiple programs, how do you even do that? **Turns out fork(2) is the way to do this. I heard about forking before, I even used it back when I was just a script kiddie. Have I fully internalised what it does or how it works? I have now!

  • **It’s fun to have a side project with a reachable goal!**I used to have side projects that have a clear goal, but sure as hell not a reachable one. I’m glad I stumbled on one that does. It sure is more motivating to finish it!

And that’s it! You can check out the project on GitHub.Also, subscribe to my blog through RSS if you can, or share my article on social media by using the buttons on the top left! 👋