Skip to content

[resolved] 507 Insufficient Storage / harddisk is not full #10268

Closed
@HerrHase

Description

@HerrHase

I have running a Server on a Ubuntu 20.04 System. It works all fine. I send files to the server,
but after uploading nearly 5 GB of Files i get this,

"507 Insufficient Storage"

I checked the space with,

Filesystem Size Used Avail Use% Mounted on
udev 999M 0 999M 0% /dev
tmpfs 204M 799k 203M 1% /run
/dev/sda1 41G 24G 15G 62% /
tmpfs 1.1G 0 1.1G 0% /dev/shm
tmpfs 5.3M 0 5.3M 0% /run/lock
tmpfs 1.1G 0 1.1G 0% /sys/fs/cgroup
/dev/sda15 63M 8.2M 55M 13% /boot/efi
tmpfs 204M 0 204M 0% /run/user/1000

I use the /dev/sda1, and there is plenty of space.

Expected Behavior

Current Behavior

Your Environment

  • Version used (minio --version): RELEASE.2020-08-13T02-39-50Z
  • Server setup and configuration: Nginx Reverse Proxy
  • Operating System and version (uname -a): Linux 5.4.0-42-generic Fix content-type in GetObjects #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Activity

harshavardhana

harshavardhana commented on Aug 14, 2020

@harshavardhana
Member

"507 Insufficient Storage"

Please provide how you are running MinIO @HerrHase like command line options etc..

HerrHase

HerrHase commented on Aug 15, 2020

@HerrHase
Author

I hope i can fill the gaps,

Minio runs with two Users, both have the same problem, i have also tried upload with the web-interface. There was no error message the upload get stuck, but on the network tab it shows me the 507.

Minio run as a Service, https://raw.githubusercontent.com/minio/minio-service/master/linux-systemd/
minio.service

Certifcates a provided by LetsEncrypt.

Configuration-Files

/etc/minio/default

MINIO_VOLUMES="/usr/local/share/minio/"
MINIO_OPTS="-C /etc/minio --address 127.0.0.1:9000"
MINIO_ACCESS_KEY=""
MINIO_SECRET_KEY=""

/etc/nginx/nginx.conf

# set user for nginx
user www-data;

# you must set worker processes based on your CPU cores
worker_processes 2;

# number of file descriptors used for nginx
# the limit for the maximum FDs on the server is usually set by the OS.
# if you don't set FD's then OS settings will be used which is by default 2000
worker_rlimit_nofile 100000;

pid /run/nginx.pid;

events {
    # determines how much clients will be served per worker
    worker_connections 1024;

    # optimized to serve many clients with each thread, essential for linux
    use epoll;

    # accept as many connections as possible
    multi_accept on;
}

http {
    server_tokens off;

    # error log will be only write from warn-level
    error_log  /var/log/nginx.error_log warn;

    # no logging for 3XX
    map $status $loggable {
        ~^[3]  0;
        default 1;
    }

    # send headers in one piece, it is better than sending them one by one
    tcp_nopush on;

    # don't buffer data sent, good for small data bursts in real time
    tcp_nodelay on;

    # caching
    # optimizes serving static files from the file system
    sendfile on;

    # assets file, 1000 files for 30 seconds
    open_file_cache max=200000 inactive=20s;
    open_file_cache_valid 30s;
    open_file_cache_min_uses 2;
    open_file_cache_errors on;

    #buffer
    client_body_buffer_size 128k;
    client_max_body_size 250m;
    client_header_buffer_size 1k;

    large_client_header_buffers 4 4k;

    output_buffers 1 32k;
    postpone_output 1460;

    # allow the server to close connection on non responding client, this will free up memory
    reset_timedout_connection on;

    # server will close connection after this time -- default 75
    keepalive_timeout 30;

    client_header_timeout 3m;
    client_body_timeout 10m;
    send_timeout 3m;

    # compress files, but not on older version of IE
    gzip on;
    gzip_min_length 1000;
    gzip_vary on;
    gzip_proxied expired no-cache no-store private auth;
    gzip_disable "MSIE [1-6]\.";
    gzip_types
        application/x-javascript
        text/css
        application/javascript
        text/javascript
        text/plain
        text/xml
        application/json
        application/vnd.ms-fontobject
        application/x-font-opentype
        application/x-font-truetype
        application/x-font-ttf
        application/xml
        font/eot
        font/opentype
        font/otf
        image/svg+xml
        image/vnd.microsoft.icon;

    # includes
    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*.conf;
    include /etc/nginx/mime.types;
}

/etc/nginx/sites-enabled/subdomain-1 (subdomain-2 has the same)

server {
    listen 80;
    listen [::]:80;
    server_name subdomain-1;
    return 301 https://subdomain-1$request_uri;
}

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    ssl_certificate /etc/letsencrypt/live/subdomain-1/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/subdomain-1/privkey.pem;

    include /etc/nginx/snippets/ssl-params.conf;
    include /etc/nginx/snippets/secure-headers.conf;

    server_name subdomain-1;

    # To allow special characters in headers
    ignore_invalid_headers off;

    # Allow any size file to be uploaded.
    # Set to a value such as 1000m; to restrict file size to a specific value
    client_max_body_size 1000m;
    
    # To disable buffering
    proxy_buffering off;

    location / {
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host $http_host;

        proxy_connect_timeout 300;

        # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        chunked_transfer_encoding off;

        proxy_pass http://127.0.0.1:9000;
    }
}
harshavardhana

harshavardhana commented on Aug 15, 2020

@harshavardhana
Member

MINIO_VOLUMES="/usr/local/share/minio/"

Can you run du -sh /usr/local/share/minio ?

HerrHase

HerrHase commented on Aug 16, 2020

@HerrHase
Author
20G	/usr/local/share/minio

On the web interface it shows me this,

Used: 12.01 GB
harshavardhana

harshavardhana commented on Aug 16, 2020

@harshavardhana
Member
20G	/usr/local/share/minio

Can you do du -sh --apparent-size and which version of MinIO is this ?

On the web interface it shows me this,

Used: 12.01 GB

Web interface shows only the usage from objects uploaded across bucket..

HerrHase

HerrHase commented on Aug 16, 2020

@HerrHase
Author
16G	/usr/local/share/minio

Version is,

minio version RELEASE.2020-08-13T02-39-50Z
added this to the Future milestone on Aug 18, 2020
adamday2

adamday2 commented on Aug 21, 2020

@adamday2

We recently ran into an issue where we ran out of inodes on our minio cache.
You may want to check with ls -i

klauspost

klauspost commented on Aug 24, 2020

@klauspost
Contributor

@HerrHase Could you check if this is a an inode issue?

HerrHase

HerrHase commented on Aug 24, 2020

@HerrHase
Author

Damn! I run out of inodes...

/dev/sda1      2493680 2493680      0  100% /

Thanks for the support! Maybe a different error message would be helpful or a hint in the documentation.

klauspost

klauspost commented on Aug 24, 2020

@klauspost
Contributor

HTTP response code 507 means "Insufficient Storage", so we have to use that.

It also seems we receive the same error from the OS in these cases, so it isn't trivial to add.

changed the title [-]507 Insufficient Storage / harddisk is not full[/-] [+][resolved] 507 Insufficient Storage / harddisk is not full[/+] on Aug 24, 2020
locked as resolved and limited conversation to collaborators on May 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Development

    No branches or pull requests

      Participants

      @vadmeste@harshavardhana@HerrHase@klauspost@adamday2

      Issue actions

        [resolved] 507 Insufficient Storage / harddisk is not full · Issue #10268 · minio/minio