One of the most important debugging tools for any application, logs tell you what went wrong and where, even if you cannot access the application or it will not deploy. It is hard to troubleshoot something that isn't there, right? Well, because logs are deployed from the moment you tell NiFi to install, that statement can be proven wrong. Logs will be generated until the point of error, and the error will in most cases point directly at a solution. NiFi provides three key types of logs.
Each NiFi server has a set of application, user and bootstrapping logs. NiFi uses SLF4J to provide a robust and configurable logging framework that you can configure to provide as much detail as you want. The logs contain detailed information about processes occurring on the server. By default, the NiFi server logs are located in the logs/ directory found in the NiFi folder. This folder is also called $NIFI_HOME. If you downloaded and untarred/unzipped NiFi, the directory is NIFI_HOME.
NiFi App Log
This is the main log file of NiFi, which logs all the activities of the Apache NiFi application, ranging from NAR files loading to run time errors or bulletins encountered by NiFi components.
Tips and tricks:
You can change your logging policies using the logback.xml available in Nifi. This will allow you to set logging policies similar to what can be done with tools like Log4J.
NiFi User Log
This log contains the user events like web security, web API config, user authorization, etc.
The appender name is USER_FILE. It follows the rollover policy. The maximum retention period for USER_FILE is 30 log files. Below are the default loggers for USER_FILE appender present in nifi-user.log.
Tips and tricks:
Again, you can set logging levels and policies.
You may also wish to specify a location for your nifi-user logs. For example, a lot of times, you may have a Kubernetes or a containerized type of environment, you want to persist that volume. In such a case you would set up a symbolic link to map that log directory, or specify the location of that log directory within your container.
NiFi Bootstrap Log
This log contains the bootstrap logs, Apache's standard output (all system.out written in the code mainly for debugging) and standard error (all system.err written in the code).
Nifi-bootstrap.log file’s appender name is BOOTSTRAP_FILE, which also follows rollback policy. The maximum retention for BOOTSTRAP_FILE appender is 5 log files. Below are the default loggers for nifi-bootstrap.log file.
Tips and tricks:
Again, you can set logging levels and policies. In this case it would be things like changing statuses from Error to Warning.
Many log errors from either the NiFi-User Log or the NiFi-App log show up in the app, within the processor generating them. (Generally not the bootstrap as that is focused on the installation.) These errors typically present as the red bulletin notifications on malfunctioning or poorly configured processors. The log errors may provide a bit more context if seen within the app, and associated with the processor that generated them. You can use the Nifi Summary to look at any processor or group of processors in the system, and drill down into the connections, the flowfiles associated, etc.
When drilling into these individual items you can also look for errors generated (which are essentially log items) to see what is happening within the context of a given processor or processor group.