Viewing cluster logs

When you encounter a problem in Virtuozzo Hybrid Infrastructure, you can send a problem report, as described in Getting technical support. The report will gather all the logs needed to troubleshoot the problem and send them to the technical support team.

Alternatively, you can investigate the root cause of your problem by using the logs listed in table below.

Service Log location Description
Metadata /vstorage/mds/logs/mds.log.zst on the storage node hosting an MDS service Storage metadata service events
Storage

/vstorage/<id>/cs/logs/cs.log.zst on the storage node hosting a CS service

To find out the log location of a particular CS on the node, run vstorage -c <cluster_name> list-services -C.

Chunk service events
Storage mountpoint /var/log/vstorage/<cluster_name>/vstorage-mount.*.blog on any storage node Software-defined storage mounting on each node
Management node /var/log/vstorage-ui-backend/messages.log and /var/log/vstorage-ui-backend/celery*.log on the management node Management node and admin panel events
/var/log/vstorage-ui-agent/* on any storage node Agent controller component events
Backup Gateway

/var/log/abgw/abgw.log*zst on any node in the Backup Gateway cluster

The latest log is abgw.log.zst, older ones are renamed to abgw.log.0.zst, abgw.log.1.zst, etc.

Backup Gateway cluster deployment and management
iSCSI /var/log/vstorage/iscsi/vstorage-target.log on any node in an iSCSI target group iSCSI target management
/var/log/vstorage/iscsi/vstorage-target-monitor.log on any node in an iSCSI target group iSCSI target monitoring
/var/log/vstorage/iscsi/scst.log.zst on any node in an iSCSI target group SCST service logs
S3 /var/log/ostor/NS-* on the S3 node with NS services S3 name server events
/var/log/ostor/OS-* on the S3 node with OS services S3 object server events
/var/log/ostor/S3GW-* on the S3 node with GW services S3 gateway events
/var/log/nginx/* on any node in the S3 cluster nginx service logs
/var/log/ostor/GR-* on the S3 node with GR services S3 geo-replicator service events
/var/log/ostor/ACC-* on the S3 node with ACC services S3 account control service events
/var/log/ostor/ostorcfgd.log*.zst on the S3 node with the Object storage configuration service (ostor-cfgd.service) Object storage configuration service events
/var/log/ostor/ostor-agent.log*.zst on any S3 node Object storage host agent service events
NFS /var/log/ganesha/ganesha.log and /var/log/ostor/ostorfs.log.gz on any node in the NFS cluster NFS server events
/var/log/vstorage/vstorage-nfsd.log on any node in the NFS cluster NFS service events
/var/log/ostor/FS-* on the node hosting an NFS share FS service events
/var/log/ostor/OS-* on the node hosting an NFS share OS service events
/var/log/ostor/ostorcfgd.log*.zst on the NFS node with the Object storage configuration service (ostor-cfgd.service) Object storage configuration service events
/var/log/ostor/ostor-agent.log*.zst on any NFS node Object storage host agent service events
Compute /var/log/vstorage-ui-backend/ansible.log on the controller node Compute cluster and add-on deployment
/var/log/hci/beholder/beholder.log on the controller node Notifications about all compute events, including VM placement

/var/log/hci/nova/* on the compute node hosting a VM

In case of problems during VM migration, view /var/log/hci/nova/nova-compute.log on the source and destination compute nodes.

Virtual machine management
/var/log/hci/neutron/neutron-l3-agent.log on any compute node Virtual routing events
/var/log/hci/neutron/neutron-openvswitch-agent.log on the compute node hosting a VM VM network interface management
/var/log/hci/cinder/* on the controller node Compute volume management
/var/log/hci/glance/glance-api.log on the controller node Image service API requests
/var/log/hci/octavia/octavia-worker.log and /var/log/hci/octavia/octavia-api.log on the controller node Load balancing service management
/var/log/hci/magnum/magnum-conductor.log, /var/log/hci/magnum/magnum-api.log, and /var/log/hci/heat/heat-engine.log on the controller node Kubernetes service and VM stack deployment and management
/var/log/hci/gnocchi/* and /var/log/hci/ceilometer/* on any compute node Billing metering service management
High availability /var/log/vstorage-ui-backend/ha.log on all management nodes High availability management
Updates /var/log/vstorage-ui-backend/software-updates.log on the management node Software update orchestration
/var/log/vstorage-ui-agent/software-updates.log on any storage node Software update downloading and installation on each node

To open log files

Use the following commands:

  • for LOG files:

    # less <log_file>.log
    
  • for BLOG files:

    # blogcat <log_file>.blog | less
    
  • for GZ files:

    # zless <log_file>.gz
    
  • for ZST files:

    # zstdless <log_file>.zst