by Don Jones

File transfers are so common today that it is easy to take the process for granted. What could go wrong with a simple file transfer operation? The answer: Plenty. For starters, there are the usual suspects: lack of disk space on the target device, insufficient privileges on the source or target directory, the source file is open for writing by another process, or the source does not even exist. Then there are the business requirement problems that do not cause technical problems but can come back to haunt you later. These include insufficient logging to satisfy compliance requirements, lack of validation to ensure the file transferred correctly, and inability to quickly and easily report on the status of a file transfer in critical operations, such as loading a data warehouse. You cannot prevent many potential problems with file transfers, but at least you can catch them early. For that, you need consolidated file transfer logging.

Syslog to the Rescue

Rather than reinvent a logging module for each of your file transfers, you can use the syslog utility. It was developed under Unix as a way of logging information in the sendmail program and proved so useful that it’s available as a generalized logging system in other programs and on other operating systems (OSs), including Windows and Linux. (It is part of Unix/Linux installs; third-party utilities are available for Windows; see http://www.syslog.org/wiki/Main/Windows.) Syslog is actually both a protocol and a set of libraries and utilities for consolidating logging messages. The simplest description is that syslog is a server that receives messages and writes them to a centralized file. This is exactly what you need to track file transfers that happen throughout the organization at different times, on different schedules, and involving different servers and a variety of trading partners.

In your file transfer programs, you can log processing information to syslog to help verify the transfer process; this log could include messages about the start and end times of a transfer, the names of files transferred, and information about the source and target devices. Just as importantly, you should log error messages if file transfers fail, the program is unable to authenticate against the target device, or there is insufficient space to store the file. Syslog has a standard message format that includes fields for:

• Facility
• Severity
• Timestamp
• Host
• Tag
• Message

Not all fields need to be used. Facility can indicate a file transfer program and severity can distinguish between information messages, such as start time of transfer, and error messages. The message can contain descriptions about the event, including error message text generated by the transfer program.

Logging information is only part of the solution: we need to get to relevant data and bypass the irrelevant messages. Fortunately, syslog files are easy to filter and there are many tools available to do so. The Syslog Wiki (http://www.syslog.org/wiki) is a great resource for tools, including filtering and analysis tools.

The original syslog protocol had reliability and security issues that have been addressed in more recent standardization effort and the development of syslog-ng, a more secure version of syslog. See http://tools.ietf.org/html/rfc5424 for more information about the protocol. For Perl developers, see the Sys:Syslog package for incorporating syslog messaging into your Perl file transfer scripts.

About the Author

Don Jones has more than a decade of professional experience in the IT industry. He's the author of more than 30 IT books, including Windows PowerShell: TFM; VBScript, WMI, and ADSI Unleashed; Managing Windows with VBScript and WMI; and many more. He's a top-rated and in-demand speaker at conferences such as Microsoft TechEd and TechMentor, and writes the monthly Windows PowerShell column for Microsoft TechNet Magazine. Don is a multiple-year recipient of Microsoft's "Most Valuable Professional" (MVP) Award with a specialization in Windows PowerShell. Don's broad IT experience includes work in the financial, telecommunications, software, manufacturing, consulting, training, and retail industries and he's one of the rare IT professionals who can not only "cross the line" between administration and software development, but also between IT workers and IT management. Don is a co-founder of Concentrated Technologies, and serves as author and series editor for Realtime Publishers.