Application Logging  and Tracing

21 April, 2022

Intro

Application logging should be consistent within the application, consistent across an organization's application portfolio and use industry standards where relevant, so the logged event data can be consumed, correlated, analyzed and managed by a wide variety of systems.

Application logs are for:

  • Providing information about problems and unusual conditions

  • Contributing additional application-specific data for incident investigation which is lacking in other log sources

  • Helping defend against vulnerability identification and exploitation through attack detection

  • Security events

  • Business process monitoring e.g. sales process abandonment, transactions, connections

  • Anti-automation monitoring

  • Audit trails e.g. data addition, modification and deletion, data exports

  • Performance monitoring e.g. data load time, page timeouts

  • Compliance monitoring

  • Data for subsequent requests for information e.g. data subject access, freedom of information, litigation, police and other regulatory investigations

  • Legally sanctioned interception of data e.g application-layer wire-tapping

  • Other business-specific requirements

What is going on? Why is it not working like that? This is not expected behavior

As a software engineer there have been situations where they are not able to figure out what is going on in the code?

Well, sometimes an Insightful log statement can come in handy to answer the queries.

Let us go over a few things which I personally have adopted and feel are decent enough for others to follow for effective and insightful logging.

Some commonly used log levels: FATAL, ERROR, WARN, INFO, DEBUG, TRACE.

Simple interpretation can be taken as:

  • FATAL: Something terribly wrong has happened, that must be investigated immediately.

  • ERROR: Something wrong has happened. Someone should take a look.

  • WARN: The process might be continued, but take extra caution.

  • INFO: Important business process has changed state/finished. We should be able to understand INFO messages and quickly find out what the application is doing.

  • DEBUG: Developer help notes.

  • TRACE: Very detailed information, intended only for development. Generally, this is for temporary items.

Implementation tips :

  • Where to record event data

  • Which events to log

  • Data to exclude

Where to record event data:

Where possible, always log:

  • Input validation failures e.g. protocol violations, unacceptable encodings, invalid parameter names and values

  • Output validation failures e.g. database record set mismatch, invalid data encoding

  • Authentication successes and failures

  • Authorization (access control) failures

  • Session management failures e.g. cookie session identification value modification

  • Application errors and system events e.g. syntax and runtime errors, connectivity problems, performance issues, third party service error messages, file system errors, file upload virus detection, configuration changes

  • Application and related systems start-ups and shut-downs, and logging initialization (starting, stopping or pausing)

  • Use of higher-risk functionality e.g. network connections, addition or deletion of users, changes to privileges, assigning users to tokens, adding or deleting tokens, use of systems administrative privileges, access by application administrators,all actions by users with administrative privileges, access to payment cardholder data, use of data encrypting keys, key changes, creation and deletion of system-level objects, data import and export including screen-based reports, submission of user-generated content - especially file uploads

  • Legal and other opt-ins e.g. permissions for mobile phone capabilities, terms of use, terms & conditions, personal data usage consent, permission to receive marketing communications

Optionally consider if the following events can be logged and whether it is desirable information:

  • Sequencing failure

  • Excessive use

  • Data changes

  • Fraud and other criminal activities

  • Suspicious, unacceptable or unexpected behavior

  • Modifications to configuration

  • Application code file and/or memory changes

Which events to log:

The level and content of security monitoring, alerting and reporting needs to be set during the requirements and design stage of projects, and should be proportionate to the information security risks. This can then be used to define what should be logged.

There is no one size fits all solution, and a blind checklist approach can lead to unnecessary "alarm fog" that means real problems go undetected.

Where possible, always log:

  • Input validation failures e.g. protocol violations, unacceptable encodings, invalid parameter names and values

  • Output validation failures e.g. database record set mismatch, invalid data encoding

  • Authentication successes and failures

  • Authorization (access control) failures

  • Session management failures e.g. cookie session identification value modification

  • Application errors and system events e.g. syntax and runtime errors, connectivity problems, performance issues, third party service error messages, file system errors, file upload virus detection, configuration changes

  • Application and related systems start-ups and shut-downs, and logging initialization (starting, stopping or pausing)

  • Use of higher-risk functionality e.g. network connections, addition or deletion of users, changes to privileges, assigning users to tokens, adding or deleting tokens, use of systems administrative privileges, access by application administrators,all actions by users with administrative privileges, access to payment cardholder data, use of data encrypting keys, key changes, creation and deletion of system-level objects, data import and export including screen-based reports, submission of user-generated content - especially file uploads

  • Legal and other opt-ins e.g. permissions for mobile phone capabilities, terms of use, terms & conditions, personal data usage consent, permission to receive marketing communications

  • Optionally consider if the following events can be logged and whether it is desirable information:

  • Sequencing failure

  • Excessive use

  • Data changes

  • Fraud and other criminal activities

  • Suspicious, unacceptable or unexpected behavior

  • Modifications to configuration

  • Application code file and/or memory changes

Event attributes:

Each log entry needs to include sufficient information for the intended subsequent monitoring and analysis. It could be full content data, but is more likely to be an extract or just summary properties.

The application logs must record "when, where, who and what" for each event.

The properties for these will be different depending on the architecture, class of application and host system/device, but often include the following:

When
  • Log date and time (international format)

  • Event date and time - the event timestamp may be different to the time of logging e.g. server logging where the client application is hosted on a remote device that is only periodically or intermittently online.

Where
  • Application identifier e.g. name and version

  • Application address e.g. cluster/hostname or server IPv4 or IPv6 address and port number, workstation identity, local device identifier

  • Service e.g. name and protocol

  • Geolocation

  • Window/form/page e.g. entry point URL and HTTP method for a web application, dialogue box name

  • Code location e.g. script name, module name

Who
  • human or machine user.

  • Source address e.g. user's device/machine identifier, user's IP address, cell/RF tower ID, mobile telephone number

  • User identity (if authenticated or otherwise known) e.g. user database table primary key value, user name, license number

What
  • Type of event

  • Severity of event Note B e.g. {0=emergency, 1=alert, ..., 7=debug}, {fatal, error, warning, info, debug, trace}

  • Security relevant event flag (if the logs contain non-security event data too)

  • Description

Data to exclude:

Never log data unless it is legally sanctioned. For example intercepting some communications, monitoring employees, and collecting some data without consent may all be illegal.

The following should not usually be recorded directly in the logs, but instead should be removed, masked, sanitized, hashed or encrypted:

  • Application source code

  • Session identification values (consider replacing with a hashed value if needed to track session specific events)

  • Access tokens

  • Sensitive personal data and some forms of personally identifiable information (PII) e.g. health, government identifiers, vulnerable people

  • Authentication passwords

  • Database connection strings

  • Encryption keys and other master secrets

  • Bank account or payment card holder data

  • Data of a higher security classification than the logging system is allowed to store

  • Commercially-sensitive information

  • Information it is illegal to collect in the relevant jurisdictions

  • Information a user has opted out of collection, or not consented to e.g. use of do not track, or where consent to collect has expired

  • Sometimes the following data can also exist, and whilst useful for subsequent investigation, it may also need to be treated in some special manner before the event is recorded:

  • File paths

  • Database connection strings

  • Internal network names and addresses

  • Non sensitive personal data (e.g. personal names, telephone numbers, email addresses)

  • Consider using personal data de-identification techniques such as deletion, scrambling or pseudonymization of direct and indirect identifiers where the individual's identity is not required, or the risk is considered too great.

  • In some systems, sanitization can be undertaken post log collection, and prior to log display.

source like : Logging - OWASP Cheat Sheet Series