Logging

We are using Symfony's Monolog bundle for logging. The configuration is located in config/packages/monolog.yaml. The other major configuration is located in App\Kernel.php. The kernel contains a method called getLogDir() which returns the path to the log directory by environment.

Environment-specific logging

The location of the log files is environment-specific.It is determined by the getLogDir() method in App\Kernel.php.

Currently, prod, stage and dev-hosted logs are located in /var/log/synqup. Other environments will write logs to /var/www/html/docker-assets/symfony/data.

Monolog configuration

The configuration file of the Monolog bundle is located in config/packages/monolog.yaml. The configuration is split into a global configuration and environment-specific configurations. The global configuration is located in the monolog key whereas the environment-specific configurations are located in when@prod, when@stage, when@dev-hosted, etc.

The list of all available Monolog handlers, formatters and processors can be found in the Monolog documentation.

MongoDB logging

The MongoDB logs are located in database logs and collection LogEntry. We are using a customer processor to log context-aware log messages. The processor is located in src/Logger/Processors/JDMProcessor.php. It extends Monolog's MongoDB formatter and adds flow execution id and JDM id to the log message.

We are also using a custom formatter to format the log messages. The formatter is located in src/Logger/Formatters/JDMMongoDBFormatter.php.

Improving the performance of MongoDB logs

It is highly recommended to use MongoDB indexes to improve the performance of MongoDB logs. The indexes that are recommended can be found in src/Documents/LogEntry.php.

If you don't need custom MongoDB indexes in your project, you can use Doctrine to create the indexes. Details can be found in MongoDB Indexes. Be aware that Doctrine will drop you custom indexes. If you have custom indexes in your project, you can create them using the MongoDB shell or the MongoDB Compass GUI.

The following indexes are recommended for the LogEntry collection:

Index	Description
{"datetime": 1}	Sorting logs by time (ascending/descending)
{"flowExecutionId": 1}, {"context.flowExecutionId": 1}	Sorting & filtering by Flow Execution. By default, FlowExecutionId is in ModuleJobDispatchContext; in synQup Core, a Monolog processor and formatter move FlowExecutionId to the top level for better indexing.
{"jobDispatcherMappingId": 1}, {"context.jobDispatcherMappingId": 1}	Sorting & filtering by JDM.
{"level_name": 1, "datetime": 1}, {"level_name": 1, "datetime": -1}	Compound index for the global log page. The first index covers filtering/sorting by level_name and sorting by datetime ascending. The second covers level_name and datetime descending. All 4 cases are covered by these 2 indexes.
{"level_name": 1, "flowExecutionId": 1, "datetime": 1}, {"level_name": 1, "flowExecutionId": 1, "datetime": -1}	Compound index for the Flow Execution log page. Same as the global log page, but here filtering is by level_name and flowExecutionId before sorting. Manual queries sorting level_name ascending and flowExecutionId descending (or vice versa) are not covered.
{"level_name": 1}	Smaller and faster index for queries not sorting by datetime.
{"level": 1}	Index for manual queries filtering by level.

It is also possible to use MongoDB's Capped collections to limit the size of the log collection. The configuration is dynamic. You can use the CLI command bin/console synqup:convert-capped-collection to set the size of the capped collection. Details can be found here.