Success factor for setting up centralized logging


The checklist of questions below will help ensure your centralized logging deployment is set up for success from the very beginning.

1. Have you implemented Network Time Protocol?
To truly enjoy the benefits of centralized logging, such as trending, correlation, and event troubleshooting, all your network elements must be running off a synchronized time source. The standard for network synchronization is Network Time Protocol (NTP).

NTP makes sure all your element logs, across disparate systems, are time-synchronized, allowing you to track down issues chronologically or create accurate time-based comparisons.

You have two choices when implementing NTP: You can download free client software or you can create your own NTP server. Free software is widely available — simply Google “NTP software” to find it.

But my advice is to take the time to create your own server that references the outside clock. This way, you avoid time being skewed if you lose connection to the outside clock for a long period of time.

2. Have you decided on your data retention policy?
How long will you store your logs? As I mentioned in my previous post, being able to perform comparative and trend analysis over days, months, or even years is one of the great benefits of centralized logs. But before you start collecting data, consider the design implications for your centralized logging server architecture.

If you’d like to be able to analyze many years of data, you’ll need to allocate sufficient disk space for storing that information. As an example, to perform a year-over-year comparison, you’ll need at least 18 months of data.

Your retention decisions may also be affected by other factors beyond your analysis needs, such as security standards or regulatory requirements for your industry. Make sure you understand the full business needs before settling on a retention policy.

3. Have you sized your database appropriately?
Your storage requirements will vary depending on the size of your network and your data retention policy, but in general, centralized logging creates large amounts of data. You can estimate the size of database you’ll need with this rough calculation:

# of servers * (size of msg * approx. # of msgs per day) * data retention period
4. Have you set event filters?
Event logs can capture a lot of data. Some of that data offers very little value and creates a lot of noise that hinders your ability to perform analysis.

It varies across systems, but generally speaking, syslog provides the following levels of events, ordered from most critical to least critical: emerg, alert, cert, err, warning, notice, info, debug.

syslog event levels: emerg, alert, cert, err, warning, notice, info, debug

I recommend you configure your elements to only send notice-level events or higher to your centralized logging server. The exception to that rule may be if you have custom applications. In that case, work with the app developers to determine what they log and how they log it so you don’t miss any critical application information.

5. Have you built redundancy into your log architecture?
Centralized logging is a crucial part of your network management toolset, so it’s extremely important to make sure it’s always available. You may even be required by law or other industry regulations to ensure no log data is lost.

Accordingly, when you’re designing your logging system, it pays to incorporate redundancy. A couple of options:

Set up two syslog servers and have all your elements send logs to both of them.
Set up two syslog servers (whose databases are replicated) behind a proxy or load balancer that’s configured to use the primary server first or switch to the secondary server if the primary fails. Remember to point your syslog clients at the proxy or load balancer IP address.
6. Have you implemented backup procedures?
Make sure you have a good backup procedure in place that includes at least full daily backups and offsite copies.

7. Have you set your security policy?
As with any core service, security should be considered before implementing a centralized log. Remember that your log server will now house all your network logs, which may include sensitive data.

Start by limiting access to the server both physically and electronically. Users should only be given access if they absolutely need it – and instances of that should be quite limited. Harden the centralized server from the operating system up, put it behind a firewall, and employ user-based access that’s tracked. I also suggest locking the server down from the outside world.

If you’re super-paranoid or some of the elements you maintain are remote, you might consider using encryption to secure the data that’s sent from the syslog client to the centralized logging server. You can do this by using a stunnel.

Now you’re primed for success! By answering these 7 questions before you deploy centralized logging, you’ll ensure your network data is secure, stable, and as helpful as possible to both your troubleshooting and planning activities.