Re: Not able to upload file >700MB file even after changing values in php.ini file


Michael C. Jaeger
 

hello,

ja a quick question: if the uploa dis 1.7Gb and the max mem limit is 1.0 GB then this would be the problem. Pls. try setting the mem limit only few GBs above post max size. (something like 2.2 GB, 2.1 GB and and 2.0 GB.

And you re right postgresql needs different setting for uploads of 1.7Gb size.

I am not sure about work meme of 2GB 8could be also Ok with 256MB), maybe you re not going to use 6GB of maintenance mem.

but a good approach is use a tool at first to get you reaosnable settings for the machine youre using:

https://pgtune.leopard.in.ua/#/

was a good help for me in the past for tuning postgresql. Please note that you re using this link at your own risk.

Kind regards, Michael

On 21. Jul 2020, at 01:12, Praba <@dpraba> wrote:

Hello,

Just started to work on Fossology tool last few days.
I am trying to upload linux5.4.30 component which is around 1.7G and not able to upload the file even after changing “post_max_size” & “upload_max_filesize”.
I was able to upload and scan other components less than 700MB.

I have Fossology installed using docker on R640 Dell server.
FOSSology version 3.8.0-86-gf0c84e229 (code revision f0c84e)

Below are the details. Please provide details if any modifications are needed.

root@abbf7f77a767:/fossology# cat /etc/php/7.0/apache2/php.ini|grep memory_limit
memory_limit = 1002M

root@abbf7f77a767:/fossology# cat /etc/php/7.0/apache2/php.ini|grep post_max_size
post_max_size = 4001M

root@abbf7f77a767:/fossology# cat /etc/php/7.0/apache2/php.ini|grep upload_max_filesize
upload_max_filesize = 3000M

Also made some tunable and config changes to few files.
/etc/sysctl.conf
kernel.shmmax=67328311296
kernel.shmall=16437576

root@abbf7f77a767:/fossology# cat /etc/postgresql//9.6/main/postgresql.conf |grep max_connections
max_connections = 100 # (change requires restart)

root@abbf7f77a767:/fossology# cat /etc/postgresql//9.6/main/postgresql.conf |grep shared_buffers
#shared_buffers = 128MB # min 128kB
shared_buffers = 32GB # min 128kB
#wal_buffers = -1 # min 32kB, -1 sets based on shared_buffers

root@abbf7f77a767:/fossology# cat /etc/postgresql//9.6/main/postgresql.conf |grep effective_cache_size
#effective_cache_size = 4GB
effective_cache_size = 10GB

root@abbf7f77a767:/fossology# cat /etc/postgresql//9.6/main/postgresql.conf |grep work_mem
#work_mem = 4MB # min 64kB
work_mem = 2GB # min 64kB
#maintenance_work_mem = 64MB # min 1MB
maintenance_work_mem = 6GB # min 1MB
#autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem

root@abbf7f77a767:/fossology# cat /etc/postgresql//9.6/main/postgresql.conf |grep maintenance_work_mem
#maintenance_work_mem = 64MB # min 1MB
maintenance_work_mem = 6GB # min 1MB
#autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem

root@abbf7f77a767:/fossology# cat /etc/postgresql//9.6/main/postgresql.conf |grep fsync
#fsync = on # flush data to disk for crash safety
fsync = on # flush data to disk for crash safety
#wal_sync_method = fsync # the default is the first option
# fsync
# fsync_writethrough
#data_sync_retry = off # retry or panic on failure to fsync

root@abbf7f77a767:/fossology# cat /etc/postgresql//9.6/main/postgresql.conf |grep full_page_writes
#full_page_writes = on # recover from partial page writes
full_page_writes = off # recover from partial page writes

root@abbf7f77a767:/fossology# cat /etc/postgresql//9.6/main/postgresql.conf |grep autovacuum
#autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem
#autovacuum = on # Enable autovacuum subprocess? 'on'
autovacuum = on # Enable autovacuum subprocess? 'on'
#log_autovacuum_min_duration = -1 # -1 disables, 0 logs all actions and
..

Thanks!

Join main@lists.fossology.org to automatically receive all group messages.