09 March, 2020

How to find out why your application suddenly closes?

Sometimes there are cases when your application just silently exits, and you have no idea why. How can you diagnose such issues?

When can a program close?

The application may close:
  1. Ordinarily: the application is closed by its own code, or the application is closed by an external process;
  2. Not ordinarily: the application's code causes a fatal error, the application is closed by the system.
Let's briefly outline the possible cases:
  • The application itself explicitly calls TerminateProcess or ExitProcess functions (directly or indirectly - for example, via Halt function);
  • The application explicitly closes/terminates all its threads (e.g. either just exited from the thread procedures or explicitly called TerminateThread or ExitThread). This usually does not happen in Delphi, because the Delphi compiler inserts a hidden call to Halt at the end of the main thread (that is, Delphi application always calls ExitProcess at the end of the main thread), but this can happen if an external process destroys the main thread in your program;
  • Some external process has closed or destroyed either your process or all threads in it;
  • An unhandled (fatal) exception occurred in your program, but the error dialog is disabled in Dr. Watson / WER (Windows Error Reporting) - either via system's settings, or via your process settings;
  • An unhandled (fatal) exception occurred in your program, a third-party post-mortem debugger with automatic start was registered in the system, and it did not display any dialog;
  • An unhandled (fatal) exception occurred in your program, which is so serious that the system could not even display the message, and the post-mortem debugger was not registered.


How to diagnose unexpected exit from the application?

Diagnostics can be carried out within the process by setting hooks to key functions:
  • Pros: you can give the application to a client, he will start it as usual, the app will do the diagnostics. Therefore, a client does not need to do anything extra. In other words, the client is not required to cooperate with you for the diagnostics;
  • Cons: not every issue can be diagnosed within the process itself. Sometimes things go haywire, so the system closes the process, process's code will have no chance to execute at all.

Diagnostics can be carried out from an external process:
  • Pros: it will be possible to diagnose almost any exit from the application;
  • Cons: the application must be run under an external process (debugger), the client will have to be instructed, he must cooperate with you.

You can use EurekaLog to diagnose both within and outside the process. You can do this without even purchasing a license. You can use a Trial edition for diagnostics from within the process, and a free EurekaLog Tools Pack for diagnostics from outside the process.

Diagnostics within the process

  1. Install EurekaLog;
  2. Enable EurekaLog for your project;
  3. In the EurekaLog's settings, go to the "Features" / "Restart&Recovery" tab and enable the "Log option application's exits" option;
  4. Launch the application and let it close;
  5. Open a folder with reports (from Windows: "Start" / "Programs" / "EurekaLog" / "EurekaLog Bug Reports"; or open %APPDATA%\Neos Eureka S.r.l\EurekaLog\Bug Reports\ folder). If you changed output path for the report file/folder - open it instead;
  6. Find a report from your application in the folder. For example: Bug Reports\Project1.exe\Project1_ExitLog.el;
  7. You can add a check for existence of this file when the application starts, and automatically send report to you;
  8. Or you can add the name of this report file to "Additional files" option, so report file will be automatically added to the next bug report.
Few notes:
  • This function will not generate an exit report for Halt (e.g. ordinarily exits);
  • The function works only for applications, not for DLLs;
  • The function will not be able to intercept exit initiated by an external process;
  • Exit reports may be false positive. For example, if the application exits via TerminateProcess function, but logically it is ordinarily exit. For example, when restarting by exception. Therefore, be careful if you want to show some kind of dialog at application's startup when the exit report is detected;
  • Technically, the feature is implemented as a wrapper for RtlReportSilentProcessExit function (only Vista +), or (Windows XP and earlier) as hooks on TerminateProcess and TerminateThread functions;
  • When you enable EurekaLog for your project, EurekaLog will be called automatically for fatal exceptions. Therefore, instead of the usual silent exit, you can get a bug report from EurekaLog.

Anyway, when you receive the report, open it as usual in the EurekaLog Viewer. The report will have a call stack at the time of exiting the application, for example:
  • ExceptionLog7.ProcessExitHandler 2244[17]
  • EInject.RtlReportSilentProcessExitHook 1166[12]
  • kernel32.TerminateProcess
  • Unit1.TForm1.Button1Click 43[1]
  • Controls.TControl.Click
  • ...


Diagnostics by an external debugger

A Threads Snapshot tool is installed together with EurekaLog, as well as with freeware EurekaLog Tools Pack. The Threads Snanshot tool is designed to capture call stacks of all threads in an application at a specific point in time.

You can register the Threads Snapshot tool as an external debugger to monitor process exits:
  1. Open a console under an administrator account at C:\Program Files (x86)\Neos Eureka Srl\EurekaLog 7\Bin\ folder (or Bin64, if you have a 64-bit application);
  2. Run:
    threadssnapshot.exe "/watch=Project1.exe"
    Where Project1.exe is the name of your application. It must be just a file name (e.g. without path). This command will register the Threads Snapshot tool to monitor exits from the specified process. Do not close the console yet, it will come in handy a bit later;
  3. Launch the application and let it close;
  4. The Threads Snapshot tool will be launched during exit. It will collect information about the exit, and ask you to save report to file after preparing it (may take a moment when there are too many threads);
  5. Run:
    threadssnapshot.exe "/unwatch=Project1.exe"
    Where Project1.exe - is exactly the same parameter that you have specified in item 2 earlier. This command will unregister monitoring.
As a result, a regular EurekaLog report will be created - in which the stack may look something like this:
  • ntdll.NtWaitForSingleObject
  • kernel32.TerminateProcess
  • Unit1.TForm1.Button1Click 43[1]
  • Controls.TControl.Click
  • ...

Technically, this functionality is implemented through Global Flags.


What else can you do?

Here is a list of things that you can try to do for additional diagnostics.

Note: in the list below, the registry key Windows Error Reporting\something indicates HKCU\Software\Microsoft\Windows\Windows Error Reporting\something registry key, and in its absence - HKLM\Software\Microsoft\Windows\Windows Error Reporting\something, or HKLM\Software\Wow6432Node\Microsoft\Windows\Windows Error Reporting\something (for 32-bit applications on a 64-bit machine).
  1. Try running the application under the debugger. Ensure that exception notifications are not disabled in the debugger's options. If the application under the debugger does not crash, or there is no way to connect the debugger, see the steps below;
  2. First of all, unregister the post-mortem debugger in the AeDebug registry key, or at least reset the Auto parameter to 0.
  3. [Vista+] Make sure that the "Windows Error Reporting Service" (WerSvc) is not disabled (e.g. it should not be in a "Disabled" state; the default startup type is "Manual", but you can run it yourself for reliability);
  4. Launch Dr. Watson (Windows 2000), report settings (Windows XP), WER settings (Windows Vista and later) - and turn ON visual alerts (Windows 2000), error reports (Windows XP), request for consent, i.e. do NOT enable automatic sending (Windows Vista and higher);
  5. [Vista+] Check WER Group Policy settings. Make sure that the UI is not disabled, logging is not disabled, consent is not set to automatically send without requests (DefaultConcent = 1). Remember to check both machine policies and user policies;
  6. [Vista+] Ensure that the Windows Error Reporting\DebugApplications\* registry key is not present or is set to 1;
  7. [Vista+] Make sure that the Windows Error Reporting\DontShowUI registry key is not present or is set to 0;
  8. [Vista+] Ensure that the Windows Error Reporting\LoggingDisabled registry key is not present or is set to 0;
  9. [Vista+] Clear all reports in system's Reliability Monitor and in "Application" system logs (so you can easily see new crash reports, if there will be any);
  10. Make sure you do not call SetErrorMode with one of the following flags: SEM_FAILCRITICALERRORS, SEM_NOGPFAULTERRORBOX, SEM_NOOPENFILEERRORBOX. For reliability, make a call to SetErrorMode(0); as first action when starting your application;
  11. [Win7+] Make sure you do not call SetThreadErrorMode with one of the following flags: SEM_FAILCRITICALERRORS, SEM_NOGPFAULTERRORBOX, SEM_NOOPENFILEERRORBOX for your threads. For reliability, make a call to SetThreadErrorMode(0); as first action in your threads;
  12. [Vista+] Make sure your code does not make a call to WerSetFlags(WER_FAULT_REPORTING_NO_UI);
  13. [Vista+] Make a call to WerSetFlags(WER_FAULT_REPORTING_ALWAYS_SHOW_UI); as first action when starting your application;
  14. Make sure that System.JITEnable global variable is set to 0. For reliability, set it to 0 as first action when you start the application;
  15. Launch your application and let it exit. If no dialog appears, then follow the steps below;
  16. Check if there are records about the application's crash in the "Applications" system log;
  17. Check for fresh entries in the system's Reliability Monitor or logs in %APPDATA%\Microsoft\Windows\WER\ReportArchive\ / %APPDATA%\CrashDumps\ folders;
  18. Try to assign your own global unhandled exception handler via the SetUnhandledExceptionFilter system function. Place a breakpoint inside your handler;
  19. Set breakpoints or hooks (within your process) to TerminateProcess, ExitProcess, and if that doesn't help, then also include TerminateThread and ExitThread functions;
  20. Set breakpoints or hooks on kernel32.KiUserExceptionDispatcher system function - if this function is called immediately before the crash/exit, then there are very high chances that your application exits because of very serious unhandled exception (for which the system could not even show a message);
  21. Finally, try setting the global hook (all processes) to TerminateProcess, TerminateThread to see if anyone else is ending your process;
  22. Also try rebuilding your application for different platform (like x86-64) or use a different version of Delphi (both newer and older). See if behaviour will be changed.
Tags: , , , , , , ,