sysadmin/nextcloud/Build.md

Setup Documentation for Nextcloud

System Setup

Configure Settings for Redis

Redis depends on a few kernel/system parameters being set appropriately to run well. Since these are kernel parameters, they can only be set on the host (not the container), and must be configured as root.

Firstly, the overcommit_memory system setting being set to zero may cause Redis to fail to background save. Make sure it is set to 1 by running:

sysctl vm.overcommit_memory=1

The setting can be made persistent through reboots by adding the line

vm.overcommit_memory = 1

to the bottom of your /etc/sysctl.conf file.

Additionally, Redis has fairly serious latency issues with THP (Transparent Huge Pages) enabled, so it is best to disable it. Disable them by running:

echo never > /sys/kernel/mm/transparent_hugepage/enabled

This configuration is also not persistent through reboots. Redis recommends adding the command as a line to your /etc/rc.local file, but rc.local is deprecated, so we will instead create a systemd unit to disable THP at startup. Create a file /etc/systemd/system/redis-disable-thp.service, with the following contents:

[Unit]
Description=Disable Transparent Huge Pages (THP) for Redis
DefaultDependencies=no
After=sysinit.target local-fs.target
Before=lxd.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/sh -c 'echo never | tee /sys/kernel/mm/transparent_hugepage/enabled > /dev/null'

[Install]
WantedBy=basic.target

Then enable the service by running:

systemctl enable redis-disable-thp

Create Container

lxc launch ubuntu-base nextcloud

Mount External Directory

inside the container, create the directory:

lxc exec nextcloud -- bash
mkdir /srv/nextcloud
cd /srv/nextcloud
mkdir database
mkdir -p files/mar/files
exit

outside the container, create the directory to be mounted, set its ownership:

cd /srv/lxd/mount
mkdir nextcloud
chown lxdfiles:lxdfiles nextcloud

create a symlink to your folder in /srv/lxd/mount, and mount the directory:

lxc config device add nextcloud database disk source=/srv/lxd/mount/nextcloud path=/srv/nextcloud/database
lxc config device add nextcloud ext-mar disk source=/tank/files/mar path=/srv/nextcloud/files/mar/files
lxc config set nextcloud raw.idmap 'both 60000 33'
lxc restart nextcloud

The rest of the setup will be done inside the container:

lxc exec nextcloud -- bash

System Setup

Install Packages

install packages required for nextcloud:

apt update
apt upgrade
apt install unzip apache2 php7.2-fpm php7.2-gd php7.2-json php7.2-pgsql php7.2-curl php7.2-mbstring php7.2-intl php-imagick php7.2-xml php7.2-zip redis-server php-redis postgresql postgresql-doc

Create Directories in external mount

create directories and set ownership:

cd /srv/nextcloud/
mkdir files
mkdir database
chown www-data:www-data files database

Download Nextcloud

Then download the zip file containing the latest version of nextcloud:

cd
wget https://download.nextcloud.com/server/releases/latest.zip

this will save it as latest.zip in your current directory.

download the checksum:

wget https://download.nextcloud.com/server/releases/latest.zip.md5

verify the checksum:

md5sum -c latest.zip.md5 < latest.zip

unzip the file:

unzip latest.zip

a nextcloud directory will appear in your root directory

Move it to /var/www/html:

mv nextcloud /var/www/html/
rm latest.zip latest.zip.md5

And finally, make sure it's owned by the www-data user:

cd /var/www/html/
chown -R www-data:www-data nextcloud

Make occ script

We will want to use the Nextcloud occ command a lot, so we will create an alias for it for convenience. Create a scipt in /usr/local/bin:

cd /usr/local/bin/
nano nc-occ

The script should have the following contents:

#!/bin/sh
sudo -u www-data php /var/www/html/nextcloud/occ "$@"

Don't forget to make it executable:

chmod +x nc-occ

Configure Apache

cd to the Apache configuration directory:

cd /etc/apache2

Activate the Apache modules required for Nextcloud:

a2enmod rewrite headers env dir mime proxy_fcgi
a2dismod php7.2
a2enconf php7.2-fpm

Next, we will create a site configuration for NextCloud. We will make a new entry in sites-available/nextcloud.conf, with the following contents:

<VirtualHost *:80>
  #ServerAdmin webmaster@localhost
  DocumentRoot /var/www/html/nextcloud/

  <Directory /var/www/html/nextcloud/>
    Require all granted
    AllowOverride All
    Options FollowSymLinks MultiViews

    <IfModule mod_dav.c>
      Dav off
    </IfModule>
  </Directory>

  ErrorLog ${APACHE_LOG_DIR}/error.log
  CustomLog ${APACHE_LOG_DIR}/access.log combined
  #LogLevel debug
  
  # PHP-FPM
  <FilesMatch "\.php$">
     SetHandler "proxy:unix:/var/run/php/php7.2-fpm.sock|fcgi://localhost/"
  </FilesMatch>
  <Proxy "fcgi://localhost/">
  </Proxy>
</VirtualHost>

Next, we will go to sites-enabled and delete the default config and make a link to our NextCloud configuration file:

cd sites-enabled
rm 000-default.conf
ln -s ../sites-available/nextcloud.conf

Configure PHP-FPM

Enable and start php-fpm:

systemctl enable php7.2-fpm
service php7.2-fpm start

Next, edit the php-fpm configuration file at /etc/php/7.2/fpm/php.ini. Find the OPcache section, which should be marked by the header [opcache], and add the following configuration:

opcache.enable=1
opcache.memory_consumption=128
opcache.interned_strings_buffer=8
opcache.max_accelerated_files=10000
opcache.revalidate_freq=2
opcache.save_comments=1

Additionally, we want to set the following parameters in the same file to increase the allowed memory limit and upload size (search the file for the parameter name and edit that line):

max_execution_time = 240
memory_limit = 1G
upload_max_filesize = 10G
post_max_size = 11G

(post_max_size should be bigger than upload_mas_filesize to prevent errors)

Finally, we will want to set some parameters to tune the performance of PHP-FPM specifically. Set the following parameters in /etc/php/7.2/fpm/pool.d/www.conf to at least the following:

clear_env = no
pm = dynamic
pm.max_children = 32
pm.start_servers = 12
pm.min_spare_servers = 8
pm.max_spare_servers = 16
pm.max_requests = 500

These parameters will depend on your particular hardware. You can use the command ps -C php-fpm7.0 -o rss= | awk '{ sum+=$1 } END { printf ("%d%s\n", sum/NR/1024,"M") }' to see the average memory usage by PHP-FPM for your computer, or ps -C php-fpm7.0 -o rss= | awk '{ sum+=$1 } END { printf ("%d%s\n", sum/1024,"M") }' to see total memory usage.

Configure Redis

We will configure Redis to be available on a unix socket as well, as that will be faster than even using a loopback interface. Edit the file /etc/redis/redis.conf, and add the following configuration options:

port 0
unixsocket /var/run/redis/redis-server.sock
unixsocketperm 770
supervised systemd

Add www-data to the redis group so it can have full permissions on the socket:

adduser www-data redis

Configure PostgreSQL

In order to configure PostgreSQL, we will switch to the postgres user:

su - postgres

First, enter the SQL shell:

psql

And create a database and user for Nextcloud:

CREATE DATABASE nextcloud;
CREATE USER ncadmin;
GRANT ALL PRIVILEGES ON DATABASE nextcloud TO ncadmin;
\q

Next, edit the file /etc/postgresql/10/main/pg_hba.conf. Since we are running nextcloud in a container, there is no need to worry about any other user connecting to the socket. So we will allow anyone to connect as ncadmin without needing authentication. So we will set the following configuration:

# Database administrative login by Unix domain socket
local   all             postgres                                peer

# Allow connection to Unix domain socket without authentication
local   all             ncadmin                                 trust

The trust directive tells PostgreSQL to allow connections to the socket without attempting to authenticate. These are the only lines that will be needed in the pg_hba.conf file. Feel free to delete or comment out all the other lines.

Exit back to being root inside the container:

exit

The last step we need is to create a script to automatically dump the database on an hourly basis. We will create a script at /usr/local/bin/dbdump, with the following contents:

#!/bin/bash

hour=$(date +%H)
day=$(date +%d)
month=$(date +%m)
year=$(date +%Y)

dbdir=/srv/nextcloud/database
dbname=nextcloud
dbuser=ncadmin

mkdir -p $dbdir
cd $dbdir

if [[ -z "$1" ]]; then
    echo "[$year-$month-$day] Error: called with missing hour parameter. Script exited without running." | tee error.log
    exit 1
fi

if [[ $hour == "$1" ]]; then
	nc-occ maintenance:mode --on
fi

pg_dump -U $dbuser -d $dbname > $dbname-hourly-$hour.sql 2>> error.log

if [[ $hour == "$1" ]]; then
	nc-occ maintenance:mode --off
	mv $dbname-hourly-$hour.sql $dbname-daily-$day.sql 2>> error.log
fi

if [[ $day == "01" ]]; then
	mv $dbname-daily-$day.sql $dbname-$year-$month-$day.sql 2>> error.log
fi

# If error.log is size 0, erase it because I don't like seeing it
if [[ ! -s ./error.log ]]; then
	rm error.log
fi

Don't forget to make it executable:

chmod +x /usr/local/bin/dbdump

Restart Services

Finally, we will restart all our services to make sure all the configuration changes have taken effect. The easiest way to do this is just restarting the container itself:

exit
lxc restart nextcloud

Nextcloud Setup

The rest of these steps should once again be executed on the nextcloud container:

lxc exec nextcloud -- bash

Install Nextcloud

We will now run the Nextcloud installation script:

nc-occ maintenance:install --data-dir "/srv/nextcloud/files/" --database "pgsql" --database-host "/var/run/postgresql" --database-name "nextcloud" --database-user "ncadmin" --database-pass "" --admin-user "${admin_username:?}" --admin-pass "${admin_password:?}"

Where ${admin_username:?} and ${admin_password:?} are chosen by you, and will be the username and password of the Nextcloud admin user that will be created.

The message Nextcloud was successfully installed should be displayed if done correctly.

Configure Nextcloud

Head over to the Nextcloud configuration directory:

cd /var/www/html/nextcloud/config/

In addition to the default config.php file, Nextcloud also loads configuration parameters fron any file ending with .config.php. These custom files are not overwritten by Nextcloud, and the values in these files take precedence over the default config file config.php, which means, among other things, that configuration set in a custom file cannot be changed through the web interface. It is also important to note that Nextcloud doesn't take these configuration values directly from the file, but rather copies them into the default config.php file. So if you want to erase a config value defined in a custom file, you will have to erase it from config.php as well.

We will create a file named my.config.php to neatly store our manually-set configuration parameters without having to search for our parameters in between all the automatically-configured parameters. The file should have the following contents:

<?php
$CONFIG = array(

/** Setup **/
'datadirectory' => '/srv/nextcloud/files/',
'htaccess.RewriteBase' => '/',

/** Database **/
'dbtype'                => 'pgsql',
'dbname'                => 'nextcloud',
'dbuser'                => 'ncadmin',
'dbpassword'            => '',
'dbhost'                => '/var/run/postgresql',
'dbtableprefix'         => 'oc_',

/** Network **/
'trusted_domains' =>
  array (
    0 => 'nextcloud.lxd',
  ),
'trusted_proxies' =>
  array (
    0 => 'nextcloud.lsu.brbytes.org',
  ),
'overwriteprotocol'     => 'http',
'overwritehost'         => 'medusa.casa.alemor.org',
'overwritewebroot'      => '/nextcloud',
'overwrite.cli.url'     => 'http://medusa.casa.alemor.org/nextcloud/',

/** Memory Caching **/
'memcache.local'        => '\\OC\\Memcache\\Redis',
'memcache.distributed'  => '\\OC\\Memcache\\Redis',
'memcache.locking'      => '\\OC\\Memcache\\Redis',
'filelocking.enabled'   => 'true',
'redis' =>
  array (
    'host' => '/var/run/redis/redis-server.sock',
    'port' => 0,
    'timeout' => 0.0,
  ),

);

Don't forget to change the owner of the file to www-data:

chown www-data:www-data my.config.php

There are also a few maintenance commands we need to run to make sure Nextcloud is properly set up:

nc-occ maintenance:update:htaccess
nc-occ db:add-missing-indices
nc-occ db:convert-filecache-bigint

Set up crontab

Nextcloud needs to execute periodic background tasks. The recommended way to do this is with cron. Open the www-data user's crontab for editing by entering:

crontab -u www-data -e

This will open a text editor where you can edit the crontab. We will add to the crontab the following lines:

# Run Nextcloud cron tasks every 5 minutes
*/5 *  *   *   *     php -f /var/www/html/nextcloud/cron.php
# Dump database every hour
01  *  *   *   *     /usr/local/bin/dbdump 01
# Scan for new files every 15 minutes
*/15 * *   *   *     /usr/local/bin/nc-occ files:scan mar

then save and exit. Finally, you can check that the job was properly scheduled by entering:

crontab -u www-data -l

Configure Reverse Proxy

The following steps need to be run on the host machine.

Configure Hosts

Find out your container's IP by using lxc list, and add it to /etc/hosts:

${container_ip:?}	nextcloud.lxd

Configure HAproxy

Add the following directives to your HAproxy configuration:

frontend www
        bind :80
        mode http

        # Reroute caldav/carddav auto-discover to appropriate address (i.e. Nextcloud)
        # Put "code 301" after "redirect" to make it a 301 redirect
        http-request redirect location /nextcloud/remote.php/dav/ if { path /.well-known/caldav /.well-known/carddav }

        # Send to nextcloud if path begins with /nextcloud
        use_backend nextcloud if { path_beg /nextcloud }

backend nextcloud
        mode http
        option forwardfor
        http-request set-header X-Forwarded-Proto https if { ssl_fc }
        # Remove the "/nextcloud" part from the path
        # WARNING: due to limitations in the config parser, the characters ")","]", or ","
        # are NOT allowed in any regular expression
        http-request set-path "%[path,regsub(^/nextcloud,)]"
        server nc-1 nextcloud.lxd:80

And make sure to reload the configuration by running:

service haproxy reload

Nextcloud Tweaks

Here are a few ways you can tweak your Nextcloud instance:

If you are using the Calendar app, the default timeframe on which it updates subscriptions is one week. To set it to update subscriptions more often, use the command:

nc-occ config:app:set dav calendarSubscriptionRefreshRate --value ${timeframe:?}

where ${timeframe:?} is in the format of a DateInterval type. For a timeframe of one hour, use PT1H. For a timeframe of one day, use P1D.

To remove the space for writing notes at the top of a folder in the Files app, run:

nc-occ config:app:set text workspace_available --value=0

It can also be disabled on a user-by-user basis by disabling the "Show rich workspaces" option in the user settings for the Files app.