Set Up Email Server on VPS with Mailcow for Full Control

June 2026 · 8 min read

Setting up your own email server on a VPS gives you complete privacy and control, but it comes with responsibilities. This guide uses Mailcow, a turnkey email stack that runs entirely through Docker with pre-configured components.

Important before starting: Running your own mail server requires an IP address that is not on any blacklist and a properly configured PTR record. Some VPS providers block port 25 by default. Check with your provider before you begin.

Why Use Mailcow

Mailcow bundles Postfix (MTA), Dovecot (IMAP), Rspamd (spam filter), SOGo (webmail), Nginx, and MariaDB into a single stack. Everything runs through Docker Compose, so you don't need to install and configure each component individually. This dramatically simplifies email server deployment.

System Requirements

Minimum 2 GB RAM (4 GB recommended)
Minimum 20 GB disk space
Ubuntu 22.04 or Debian 11/12
Docker and Docker Compose already installed
A domain with DNS control

Installing Mailcow

cd /opt
git clone https://github.com/mailcow/mailcow-dockerized
cd mailcow-dockerized
./generate_config.sh   # Will prompt for FQDN like mail.example.com
docker compose pull
docker compose up -d

After installation, open https://mail.example.com to access the admin panel. Default credentials are username admin and password moohoo. Change these immediately.

Required DNS Records

# A Record for mail server
mail.example.com    A     1.2.3.4

# MX Record pointing to mail server
example.com         MX    10  mail.example.com.

# SPF: Authorizes mail server to send email
example.com         TXT   "v=spf1 mx ~all"

# DKIM: Mailcow generates the key, copy from admin panel
mail._domainkey     TXT   "v=DKIM1; k=rsa; p=..."

# DMARC
_dmarc.example.com  TXT   "v=DMARC1; p=quarantine; rua=mailto:[email protected]"

# Autoconfig for email clients
autoconfig          CNAME mail.example.com.
autodiscover        CNAME mail.example.com.

PTR Record: Critical for Mail Server Success

Set this through your VPS provider, not through the DNS panel. Your VPS IP must have a reverse DNS (PTR) record pointing back to mail.example.com. Without a proper PTR record, most mail servers will immediately reject your emails.

Testing Your Setup

Send a test email to Gmail or Outlook and check if it lands in spam
Check the DKIM signature in the email headers of received messages
Use mail-tester.com to score your email — aim for 9/10 or higher

Alternative: Use Email Hosting Instead

Running your own mail server requires continuous maintenance: updates, blacklist monitoring, and troubleshooting deliverability issues. If you don't want to manage all of this yourself, email hosting services are a reasonable alternative.

For businesses in Thailand, AsiaGB's hosting services include email hosting, spam filtering, and Thai-language support — ideal for teams that don't want to maintain their own mail server.

Before setting up or migrating your email server, always verify domain health and DNS records with a tool like dnsxray.com, which checks MX, SPF, DKIM, and security headers for free in a single interface.

Summary

Mailcow makes it much easier to set up an email server on a VPS, but the real complexity lies in DNS configuration and email deliverability — not installation. If you're prepared to maintain the system yourself, it's worth trying. But if you value convenience, email hosting is a solid alternative.

Verify the System is Working Correctly

# Check all containers are running
cd /opt/mailcow-dockerized
docker compose ps

# View mail logs
docker compose logs postfix-mailcow --tail=50

# Check DKIM records
dig TXT mail._domainkey.example.com +short

Test score: Send an email to [email protected] and you'll receive an automatic reply with SPF/DKIM/DMARC verification results.

Common Issues & Troubleshooting

Error: Port 25 connection refused / Emails won't send

Many VPS providers block port 25 by default. You'll need to request it to be opened, or use a relay service:

# Test if port 25 is open
telnet mail.example.com 25
# If "connection refused," it's blocked at the network level

# Alternative: Send via port 587 (submission) instead
# Or use a relay through SendGrid/Mailgun for outbound email

Emails land in Spam at Gmail / Outlook

The main causes are IP blacklist or incorrect PTR record:

# Check your PTR record
dig -x YOUR_VPS_IP +short
# Should return mail.example.com.

# Check if your IP is on blacklist
# Use the website mxtoolbox.com/blacklists.aspx

Error: DKIM signature fails in email headers

DKIM key in DNS hasn't updated yet or was copied incorrectly:

# Get DKIM key from Mailcow admin panel
# Email → Configuration → Domain → DKIM

# Verify DNS has propagated
dig TXT mail._domainkey.example.com
# Should show p=MIIBIj... (RSA public key)

Error: Docker container restart loop

# View logs of the problematic container
docker compose logs dovecot-mailcow --tail=100

# Restart a single container
docker compose restart dovecot-mailcow

# Pull new images
docker compose pull && docker compose up -d

Webmail won't load / SSL error

# Run Let's Encrypt renewal
docker compose exec acme-mailcow /root/start.sh

# Check certificate expiration
echo | openssl s_client -connect mail.example.com:443 2>/dev/null | openssl x509 -noout -dates

Frequently Asked Questions

Can I host multiple domains on a single mail server?

Yes. Mailcow supports unlimited multi-domain hosting. Add a new domain through the admin panel and configure DNS records separately for each domain.

How much resources does Mailcow need?

Minimum 2 GB RAM, though 4 GB is recommended for production use. Disk space depends on email volume — 40 GB+ is suggested, plus additional space for backups depending on your retention needs.

How do I backup email?

cd /opt/mailcow-dockerized
./helper-scripts/backup_and_restore.sh backup all
# Backups are stored in /opt/mailcow-dockerized/backups/

How do I update Mailcow?

cd /opt/mailcow-dockerized
./update.sh

Always backup before updating, and review the changelog for any breaking changes.