Skip to content

Latest commit

 

History

History
106 lines (59 loc) · 4.02 KB

11.md

File metadata and controls

106 lines (59 loc) · 4.02 KB
Raw

+++ date = "2015-09-30" draft = false weight = 11 title = "Lab 11 - Logs" +++

Lab Duration: 10 minutes

Lab Objective

The objective of this lab is to show how to locate the various logs for services supporting OpenStack, thereby allowing admins to monitor their OpenStack deployment. The second part of this lab is designed as a reference for how to build an rsyslog server for your OpenStack infastruture. This lab has no dependency on other labs.

1. Where are the logs?

Most OpenStack services use the convention of writing their log files to sub directories of /var/log/

  1. Display the nova-* service logs

    [root@controller ~]# ls /var/log/nova

  2. Display the glance-* service logs

    [root@controller ~]# ls /var/log/glance

  3. Display the cinder-* service logs

    [root@controller ~]# ls /var/log/cinder

  4. Display the keystone-* service logs

    [root@controller ~]# ls /var/log/keystone

  5. Display the horizon service logs

    [root@controller ~]# ls /var/log/horizon

  6. Boot up messages for VM instances (on compute nodes)

    • First SSH to a compute node (the host names of compute are compute1 and compute2)

    [root@compute1 ~]# cat /var/lib/nova/instances/instance-<instance id>/console.log

  7. Display cinder-volume logs (on block storage nodes)

    [root@compute1 ~]# cat /var/log/cinder/cinder-volume.log

2. Introduction to rsyslog (or remote logging) (Reference)

All systems generate and update log files recording their actions and any problems they encounter. In a distributed or cloud computing environment that contains many systems, and it may be unreasonable to expect admins to collect log files from these many locations. Remote logging is the solution to this problem.

The rsyslog service provides a centralized logging server, and configuration for individual systems to send their log files to the centralized logging server.

  1. We won't actually do these steps, but it might be useful to see how they would be performed in a production system.

  2. Firstly, install the rsyslog server on the system you want to use as the centralized logging server. All other nodes will be configured to send logs to this location, and they too will need rsyslog installed. We'll assume an installation onto a CentOS system (this is what you're using in class)

    • # yum install rsyslog

    • To be clear, you'd install rsyslog on the centralize logging server, as well as every node you wish to have report logging.

  3. Allow SELinux to allow rsyslog traffic (centralized server only)

    • # semanage -a -t syslogd_port_t -p udp 514

  4. Configure /etc/sysconfig/iptables to allow rsyslog traffic, then restart the service so the new rules take effect (centralized server only)

    • Note -- IP TABLES ACCEPT rules need to appear before REJECT rules

    • # vim /etc/sysconfig/iptables

    • -A INPUT -m state --state NEW -m udp -p udp --dport 514 -j ACCEPT <- place this line inside of /etc/sysconfig/iptables

    • # service iptables restart

  5. Configure /etc/rsyslog.conf to define where the logs will be saved to. Place the following snippet in rsyslog.conf (centralized server only)

    • # vim /etc/sysconfig/iptables
    $template TmplAuth, "/var/log/%HOSTNAME%/%PROGRAMNAME%.log" 

authpriv.*   ?TmplAuth
*.info,mail.none,authpriv.none,cron.none   ?TmplMsg

  6. Before you exit and save /etc/rsyslog.conf, uncomment the following lines

    $ModLoad imudp
$UDPServerRun 514

  7. The central server is now configured-- time to turn our attention to the individual nodes.

  8. On each node (not the centralized server), edit the /etc/rsyslog.conf, and specify the address of your centeralized logging server with the following:

    • *.* @REPLACE_WITH_IP_ADDRESS_OF_SERVER:PORT_SERVER_LISTENING_ON

    • The port used in the above statement would be 514 if you are following this procedure as it is written

  9. Start rsyslog, and then confirm that rsyslog will always run on startup (do this on the centralized server AND all nodes)

    • # service rsyslog start

    • # chkconfig rsyslog on