Self-Hosted Cloud Environment (Search Engine)

In this article we are going to setup our first self-hosted web application called SearXNG. This will allow us to replace Google, Duck Duck Go, Bing and the like. SearXNG will search all of these services and more on our behalf, while also cleaning up and removing the tracking information as well.

This post is but one in a series for setting up your own cloud environment.

If you wish to have your own hosted personal search engine continue on to the following steps, if you do not you can skip this application as our future installs will not rely on this install.

First we will create a folder on the host system to store some static files that will survive updating the instance in the future. SSH into your VPS and execute the following commands:

username@vps-8734:~$ sudo -i
root@vps-8734:~# mkdir -p /media/searxng

Now lets switch back to our Portainer web administration interface and click “Containers” on the left and “Add container” at the top. Starting at the top of the page for the name lets use “searxng” and for the image go ahead and put in “searxng/searxng:latest”.

Click the gray “publish a new network port” button and fill out the fields as such:

Host: 6081 -> Container: 8080

Now at the bottom click on “Restart policy” and change it to “Always”.

Now lets setup some environmental variables, these are settings that will be pushed into the docker instance when we deploy the container. Click on “Env” at the bottom of the page.

Name: BASE_URL -> Value: https://search.<yourdomain.xyz>/
Name: INSTANCE_NAME -> Value: searxng
Name: SEARXNG_SETTINGS_PATH -> Value: /etc/searxng/settings.yml
Name: UWSGI_SETTINGS_PATH -> Value: /etc/searxng/uwsgi.ini

In the above block you will want to replace <yourdomain.xyz> with your own domain, I suggest using a subdomain like “search.yourdomain.xyz” here or maybe “search.apps.yourdomain.xyz”. Just keep all your future apps following the same standard as it will keep your DNS records clean.

Now skip over to “Volumes” and click the gray “map additional volume” button. Create the following mapping, and change the type on the right to “Bind”.

Container: /etc/searxng -> Host: /media/searxng

Go ahead and deploy the container using the blue “Deploy the container” button.

Now we have the instance running, but we need to get a few things setup. First we need to get our DNS hostname working we created under the environment variables. Second we need to setup this instance inside our reverse proxy so we can get secure HTTPS access to the instance.

DNS Settings / Hostname

This process will differ for everyone, as this portion depends where your domain is registered and which NS servers you are using for your domain. Here is where you need to check your domain registrar (something like NameCheap, GoDaddy, DirectNIC, NetworkSolutions, etc) and find your DNS records.

I am going to create two records, first I am going to make an “A” record. This record points a hostname using your domain, something like docker.yourdomain.xyz, and points to the public IP address of your VPS.

Once your “A” record is created, create another record this time a “CNAME” and for the name put in the value of your “BASE_URL” parameter you used in the environmental variables section of Portainer. In our example I will be using “search.yourdomain.xyz” and for the destination or pointer you should use your docker “A” record you created previously. So the following is what the two records should look somewhat like the following:

Record: A
Hostname: docker
Domain: yourdomain.xyz

Record: CNAME
Hostname: search
Pointer/Dest: docker.yourdomain.xyz

Once you have those created we need to have our reverse proxy answer those requests.

Login to your Nginx Proxy Manager administrative interface, we set this up back in the second article in this series. Once you are logged into the UI, click on “Hosts” at the top then “Proxy Hosts” in the drop down menu. Now click on the “Add Proxy Host” button in the top right.

For our domain name, this is the FQDN (Fully Qualified Domain Name) that the reverse proxy will listen for on incoming connections. Set this to “search.yourdomain.xyz”, leave the scheme as “http”, set the “Forward Hostname / IP” to “docker.yourdomain.xyz”, and set the forward port to “6081”. You can also set the slider for “Block Common Exploits” to enabled.

Click Save at the bottom right and the dialog will go away, and a new Proxy Host will appear in the list. Currently running as HTTP Only.

Checking Local DNS Resolution

I want to pause here and recommend we check the DNS settings on your VPS. We want to make sure the hosts file contains an entry for “docker.yourdomain.xyz” as this will make your local DNS lookups much faster.

In your SSH session to your VPS execute the following command, you should already be root from the previous commands we ran at the top of this post:

root@vps-8734:~# nano /etc/hosts

If you do not have a line in this file that looks like the following, just add it.

<publicipaddress>    docker.yourdomain.xyz

Remember to replace “<publicipaddress>” with your public IP address of the VPS and “yourdomain.xyz” with your domain name.

Getting your HTTPS Certificate and Verifying Traffic

This step will verify a few things and if all traffic is flowing correctly, give you a free SSL/TLS certificate from Lets’ Encrypt.

Click the three dots on the right of the entry for “search.yourdomain.xyz” and choose “Edit”. Click “SSL” on the top menu bar and select “Request a new SSL certificate” from the drop down. I also usually choose “Force SSL” and “HTTP/2 Support”. At the bottom verify the email is good (they use this in case the certificate is expiring and no renew has happened to inform you), and choose “I agree”.

Click the “Save” button and give it a little bit of time. If the dialog goes away without errors you have successfully pulled your first SSL certificate. If you got any errors, then you need to verify a few things.

  • Firewalls; Are you allowing ports 80 and 443 through your firewalls to the reverse proxy docker instance?
  • Are all your DNS records correct?
    • Verify your “A” record that points your hostname of “docker.yourdomain.xyz” to the public IP address of your docker server.
    • Verify your “CNAME” record points “search.yourdomain.xyz” to “docker.yourdomain.xyz”.

You can retry five times before you will be temp banned from Lets’ Encrypt for that FQDN and will need to wait an hour to try again. Don’t keep at it if it fails, check all your settings and give your DNS provider some time to public the changed records.

Does it work?

If everything is working, you should be able to open your browser and navigate to:

https://search.yourdomain.xyz

At this point you should get to a default SearXNG page, ready for you to set preferences or start searching.

2 thoughts on “Self-Hosted Cloud Environment (Search Engine)

Leave a Reply

Your email address will not be published.