Sstrhsrtj

The server in question is used for processing data into reports, it has three cron jobs which execute every minute to check the process queue and if anything is found they'll keep running until the queue is empty. On particularly busy days (every Tuesday for the last 3 months and other odd days) the server crashes.

By running ps aux | grep 'httpd' | awk '{print $6/1024 " MB";}' I can see the three httpd workers are consistently gaining memory in a linear fashion. Since there are tons of reports the processes don't end and they continue to absorb more memory.

This is a PHP runtime, checking memory usage inside there we can see it doesn't get anywhere near being an issue. There's a hard PHP memory limit of 256MB and in the PHP layer at least the memory is released after a report is finished processing. Currently using a PHP memory profiler on a duplicate server to see if there are any issues in the PHP layer causing this. However the ps aux command mentioned above shows memory starting at about 270MB for a httpd worker and rising and rising until the whole server (8GB RAM) runs out. So it seems likely to me that httpd is holding onto all memory the PHP layer is using and not recycling it back into the PHP process or into the system. Snippet from the error log when this occurs below.

[Tue Dec 04 09:20:34.805175 2018] [mpm_prefork:error] [pid 2751] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process

[Tue Dec 04 09:20:45.142735 2018] [mpm_prefork:error] [pid 2751] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process



mmap() failed: [

mmap() failed: [

mmap() failed: [1122]]  CCannot allocate memoryannot allocate memory12] Cannot allocate memory





mmap() failed: [

mmap() failed: [1122]]  CCannot allocate memoryannot allocate memory



mmap() failed: [12] Cannot allocate memory[Tue Dec 04 09:20:45.764357 2018] [php7:error] [pid 12198] [client 127.0.0.1:37694] PHP Fatal error:

  Out of memory (allocated 2097152) (tried to allocate 65536 bytes) in /var/www/html/vendor/composer/ClassLoader.php on line 440

[Tue Dec 04 09:20:45.764411 2018] [php7:error] [pid 12197] [client 127.0.0.1:37692] PHP Fatal error:  Out of memory (allocated 2097152) (tried

 to allocate 65536 bytes) in /var/www/html/vendor/composer/ClassLoader.php on line 440

[Tue Dec 04 09:20:45.764358 2018] [php7:error] [pid 12191] [client 127.0.0.1:37690] PHP Fatal error:  Out of memory (allocated 2097152) (tried

 to allocate 65536 bytes) in /var/www/html/vendor/composer/ClassLoader.php on line 440







mmap() failed: [

mmap() failed: [1122]]  CCannot allocate memoryannot allocate memory



mmap() failed: [12] Cannot allocate memory





mmap() failed: [

mmap() failed: [1122]]  CCannot allocate memoryannot allocate memory



mmap() failed: [12] Cannot allocate memory[Tue Dec 04 09:20:46.257438 2018] [php7:error] [pid 12191] [client 127.0.0.1:37690] PHP Fatal error:

  Out of memory (allocated 2097152) (tried to allocate 73728 bytes) in /var/www/html/vendor/composer/ClassLoader.php on line 440

[Tue Dec 04 09:20:46.257439 2018] [php7:error] [pid 12197] [client 127.0.0.1:37692] PHP Fatal error:  Out of memory (allocated 2097152) (tried

 to allocate 73728 bytes) in /var/www/html/vendor/composer/ClassLoader.php on line 440

[Tue Dec 04 09:20:46.257481 2018] [php7:error] [pid 12198] [client 127.0.0.1:37694] PHP Fatal error:  Out of memory (allocated 2097152) (tried

 to allocate 73728 bytes) in /var/www/html/vendor/composer/ClassLoader.php on line 440

[Tue Dec 04 09:20:57.249945 2018] [mpm_prefork:error] [pid 2751] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process

[Tue Dec 04 09:23:08.171314 2018] [mpm_prefork:error] [pid 2751] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process

[Tue Dec 04 09:24:33.415351 2018] [mpm_prefork:error] [pid 2751] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process

[Tue Dec 04 09:26:22.308600 2018] [mpm_prefork:error] [pid 2751] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process

[Tue Dec 04 09:27:55.057324 2018] [mpm_prefork:error] [pid 2751] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process

[Tue Dec 04 09:29:17.174173 2018] [mpm_prefork:error] [pid 2751] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process

[Tue Dec 04 09:30:39.193341 2018] [mpm_prefork:error] [pid 2751] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process

[Tue Dec 04 09:33:16.023329 2018] [mpm_prefork:error] [pid 2751] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process

[Tue Dec 04 09:34:53.208958 2018] [mpm_prefork:error] [pid 2751] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process

[Tue Dec 04 09:35:30.902310 2018] [mpm_prefork:error] [pid 2751] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process

[Tue Dec 04 09:36:05.215192 2018] [mpm_prefork:error] [pid 2751] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process

[Tue Dec 04 09:37:36.511811 2018] [mpm_prefork:error] [pid 2751] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process

[Tue Dec 04 09:42:18.453045 2018] [suexec:notice] [pid 2699] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)

httpd mods in use:

$ sudo httpd -M

Loaded Modules:

 core_module (static)

 so_module (static)

 http_module (static)

 access_compat_module (shared)

 actions_module (shared)

 alias_module (shared)

 allowmethods_module (shared)

 auth_basic_module (shared)

 auth_digest_module (shared)

 authn_anon_module (shared)

 authn_core_module (shared)

 authn_dbd_module (shared)

 authn_dbm_module (shared)

 authn_file_module (shared)

 authn_socache_module (shared)

 authz_core_module (shared)

 authz_dbd_module (shared)

 authz_dbm_module (shared)

 authz_groupfile_module (shared)

 authz_host_module (shared)

 authz_owner_module (shared)

 authz_user_module (shared)

 autoindex_module (shared)

 cache_module (shared)

 cache_disk_module (shared)

 cache_socache_module (shared)

 data_module (shared)

 dbd_module (shared)

 deflate_module (shared)

 dir_module (shared)

 dumpio_module (shared)

 echo_module (shared)

 env_module (shared)

 expires_module (shared)

 ext_filter_module (shared)

 filter_module (shared)

 headers_module (shared)

 http2_module (shared)

 include_module (shared)

 info_module (shared)

 log_config_module (shared)

 logio_module (shared)

 macro_module (shared)

 mime_magic_module (shared)

 mime_module (shared)

 negotiation_module (shared)

 remoteip_module (shared)

 reqtimeout_module (shared)

 request_module (shared)

 rewrite_module (shared)

 setenvif_module (shared)

 slotmem_plain_module (shared)

 slotmem_shm_module (shared)

 socache_dbm_module (shared)

 socache_memcache_module (shared)

 socache_shmcb_module (shared)

 status_module (shared)

 substitute_module (shared)

 suexec_module (shared)

 unixd_module (shared)

 userdir_module (shared)

 version_module (shared)

 vhost_alias_module (shared)

 watchdog_module (shared)

 dav_module (shared)

 dav_fs_module (shared)

 dav_lock_module (shared)

 lua_module (shared)

 mpm_prefork_module (shared)

 proxy_module (shared)

 lbmethod_bybusyness_module (shared)

 lbmethod_byrequests_module (shared)

 lbmethod_bytraffic_module (shared)

 lbmethod_heartbeat_module (shared)

 proxy_ajp_module (shared)

 proxy_balancer_module (shared)

 proxy_connect_module (shared)

 proxy_express_module (shared)

 proxy_fcgi_module (shared)

 proxy_fdpass_module (shared)

 proxy_ftp_module (shared)

 proxy_http_module (shared)

 proxy_hcheck_module (shared)

 proxy_scgi_module (shared)

 proxy_uwsgi_module (shared)

 proxy_wstunnel_module (shared)

 ssl_module (shared)

 php7_module (shared)

$ httpd -l

Compiled in modules:

  core.c

  mod_so.c

  http_core.c

httpd/http.d/http.conf

<IfModule mpm_prefork_module>

    StartServers             2

    MinSpareServers          2

    MaxSpareServers          4

    MaxRequestWorkers        10

    MaxConnectionsPerChild 1

</IfModule>

... three cron jobs which execute every minute to check the process queue and if anything is found they'll keep running until the queue is empty

This is prone to memory leaking so instead use a bash demon to sequentially spawn a new instance for every item in the queue;

while true ; do

    while read -r F ; do

        php /var/www/html/my.site.com/process1.php "$F"

    done < <(find /path/to/queue -type f)

    sleep 1

done

calling

wget -q -O /dev/null http://localhost/core/processing

is needlessly involving apache and allowing it to hold on to memory for more than the duration of one item.