[superm1]

1. replaces all std::cout/std::cerr with structured LOG(SEVERITY, "Tag") calls
2. Cleans up noisy messages — downgrades many INFO messages to DEBUG, removes decorative separators
3. Adds new CLI features — --host/--port on 5 subcommands (list, pull, delete, status, stop)
4. Rewrites Linux server discovery — uses pgrep/lsof instead of PID file
5. Rewrites systemd detection — adds INVOCATION_ID check and isatty() guard
6. Suppresses subprocess output — redirects to /dev/null or NUL when not in debug mode
7. Fixese a bug in crouter.cpp (log_level_→blog_level_)

[superm1]

Here is an updated view of startup (before you launch a model).

╰─❮ ./build/lemonade-server serve
[ServerManager] PID file created: /tmp/lemonade-router.pid
Lemonade Server v9.4.0 started on port 8000
API endpoint: http://127.0.0.1:8000/api/v1
Connect your apps to the endpoint above.
Documentation: https://lemonade-server.ai/
[ServerManager] PID file created: /tmp/lemonade-router.pid
Press Ctrl+C to stop

Here is under systemd:

Feb 26 08:56:35 lilac-hi systemd[1]: Started Lemonade Server.
Feb 26 08:56:35 lilac-hi lemonade-server[194058]: Starting Lemonade Server...
Feb 26 08:56:35 lilac-hi lemonade-server[194058]:   Version: 9.4.0
Feb 26 08:56:35 lilac-hi lemonade-server[194058]:   Port: 8000
Feb 26 08:56:35 lilac-hi lemonade-server[194058]:   Host: localhost
Feb 26 08:56:35 lilac-hi lemonade-server[194058]:   Log level: info
Feb 26 08:56:35 lilac-hi lemonade-server[194058]: [Server] Detected systemd environment - will use journald for log streaming
Feb 26 08:56:35 lilac-hi lemonade-server[194058]: [Server] [Net Broadcast] Unable to broadcast my existance please use a RFC1918 IPv4,
Feb 26 08:56:35 lilac-hi lemonade-server[194058]: [Server] [Net Broadcast] or hostname that resolves to RFC1918 IPv4.
Feb 26 08:56:36 lilac-hi lemonade-server[194058]: [ModelManager] Building models cache...
Feb 26 08:56:36 lilac-hi lemonade-server[194058]: [ModelManager] Backend availability:
Feb 26 08:56:36 lilac-hi lemonade-server[194058]:   - NPU hardware: Yes
Feb 26 08:56:36 lilac-hi lemonade-server[194058]:   - System RAM: 26.7 GB (max model size: 21.3 GB)
Feb 26 08:56:36 lilac-hi lemonade-server[194058]:   - Largest memory pool: 13.3
Feb 26 08:56:36 lilac-hi lemonade-server[194058]: [ModelManager] Cache built: 52 total, 0 downloaded
Feb 26 08:56:47 lilac-hi lemonade-server[194058]: [Server] Shutdown signal received, exiting...
Feb 26 08:56:47 lilac-hi lemonade-server[194022]: Received interrupt signal, shutting down...
Feb 26 08:56:47 lilac-hi lemonade-server[194022]: Shutting down server...
Feb 26 08:56:47 lilac-hi systemd[1]: Stopping Lemonade Server...

[superm1]

And here is what model load/unload looks like now.

[Server] Loading model: Gemma-3-4b-it-GGUF 
[Router] Loading model: Gemma-3-4b-it-GGUF...
llama-server is ready!
[Router] Model loaded successfully. Total loaded: 1
[Server] POST /api/v1/load - 200
[Server] Unload request received
[Server] Request method: POST, body length: 35
[Server] Content-Type: application/json
[Router] Unload model called: Gemma-3-4b-it-GGUF
[Router] Evicting model: Gemma-3-4b-it-GGUF
[ProcessManager] Process terminated, waiting for GPU driver cleanup...
[Router] Evicted model: Gemma-3-4b-it-GGUF
[Server] Model 'Gemma-3-4b-it-GGUF' unloaded successfully
[Server] POST /api/v1/unload - 200

[bitgamma]

looks good to me, but we really need to consider moving to a proper logging framework or we will drown in log-level checks and log-level parameter passed in every function

[jeremyfowers]

we will drown in log-level checks and log-level parameter passed in every function

reading the change set I feel this

[superm1]

looks good to me, but we really need to consider moving to a proper logging framework or we will drown in log-level checks and log-level parameter passed in every function

Any suggestions for a particular cross platform framework?

[jeremyfowers]

My design philosophy for logging is that the user can't be trusted to set the right log level before they have a problem, so the only supported log level should be debug.

I'm wary of any logging choice that would allow users to hide debug info.

I'm also wary of the sheer amount of changes here: devs now need to remember to add a if (log_level_ == "debug" || log_level_ == "trace") virtually every time they add a cout? Is there not a more streamlined way to accomplish the same goal?

[bitgamma]

haven't used any recently, but this looks pretty straightforward: https://github.com/badaix/aixlog

[bitgamma]

while I agree with you in principle, for a long running server instance the log file can hog the system (and kill SSDs long term) so by default the logs should be minimal.

I feel this shows we need a logging framework that makes this much more manageable though

My design philosophy for logging is that the user can't be trusted to set the right log level before they have a problem, so the only supported log level should be debug.

I'm wary of any logging choice that would allow users to hide debug info.

I'm also wary of the sheer amount of changes here: devs now need to remember to add a if (log_level_ == "debug" || log_level_ == "trace") virtually every time they add a cout? Is there not a more streamlined way to accomplish the same goal?

[superm1]

My design philosophy for logging is that the user can't be trusted to set the right log level before they have a problem, so the only supported log level should be debug.

As @bitgamma points out this will fill up SSDs really quick. Most software runs at INFO by default and has a mechanism to increase it if a problem comes up. I don't think we should be straying from that.

I'm also wary of the sheer amount of changes here

Let me look at switching to a logging framework. It's unfortunate I found bugs in the process of doing this that are fixed and we should get in, but hopefully switching to a framework isn't too much difficulty.

[superm1]

I integrated that framework suggested by @bitgamma . Have a look again. I personally like this a lot better.

[superm1]

I need to double check all the places that log_level was plumbed earlier, so I will double check those next.

[bitgamma]

I'll wait until you are done with removing the log_level argument from all functions before reviewing, but this is looking good.

[superm1]

I'll wait until you are done with removing the log_level argument from all functions before reviewing, but this is looking good.

Should be good to go now, it's rebased on the other big one that just merged (flm-groundwork)

[bitgamma]

looks good to me. I removed a couple of left over instances of unused log_level, I hope you don't mind.

[jeremyfowers]

If this PR does all of these things please update the title and description:

1. Introduces the aixlog logging framework — replaces all std::cout/std::cerr with structured LOG(SEVERITY, "Tag") calls 2. Cleans up noisy messages — downgrades many INFO messages to DEBUG, removes decorative separators 3. Adds new CLI features — --host/--port on 5 subcommands (list, pull, delete, status, stop) 4. Rewrites Linux server discovery — uses pgrep/lsof instead of PID file 5. Rewrites systemd detection — adds INVOCATION_ID check and isatty() guard 6. Suppresses subprocess output — redirects to /dev/null or NUL when not in debug mode 8. Fixesea bugaincrouter.cppv(log_level_→blog_level_)ormat change

[jeremyfowers]

claude /review found some critical-level and moderate-level issues, thoughts?

Critical Issues

  1. ~19 instances of missing spaces in error messages (server.cpp)

  The migration introduced broken error strings like:
  "ERROR inhandle_completions"    // should be "ERROR in handle_completions"
  "JSON parse error inhandle_image_edits"
  This affects nearly every error handler in server.cpp. Clearly a find-and-replace artifact.

  2. #pragma once removed from single_instance.h

  The include guard was removed with no #ifndef replacement. This will cause multiple-definition errors if included from more than one translation unit.

  3. LOG() in signal handler (main.cpp)

  void signal_handler(int signal) {
      LOG(INFO) << "Shutdown signal received, exiting..." << std::endl;

  LOG() involves mutexes and memory allocation — not async-signal-safe. Use write(STDOUT_FILENO, ...) or keep the old std::cout (also technically unsafe but simpler).

  4. LOG() after fork() in child process (process_manager.cpp)

  After fork(), using LOG() (which holds mutexes cloned from the parent) is unsafe in multi-threaded programs.

  5. Doubled tags in log messages

  Several log messages now have redundant tags:
  LOG(ERROR, "FastFlowLM") << "[FastFlowLM ERROR] Failed to download..."
  LOG(DEBUG, "LlamaCpp") << "[LlamaCpp] Model loaded on port..."
  LOG(ERROR, "WrappedServer") << "[WrappedServer] Backend returned HTTP..."

  ---
  Moderate Issues

  6. Inconsistent timestamp format
  - main.cpp: "%Y-%m-%d %H:%M:%S.#ms" (colons)
  - tray/main.cpp: "%Y-%m-%d %H-%M-%S.#ms" (dashes)

  7. is_quiet_get — dead code (server.cpp)

  A variable is computed to detect low-signal GET requests but is never used. Looks like an incomplete implementation.

  8. Leftover std::cerr in fastflowlm_server.cpp

  Two std::cerr calls were not converted to LOG() right after a LOG(ERROR) call.

  9. Redundant "Warning:" prefix

  Several messages now read [WARNING] (ModelManager) Warning: Could not load... — double "warning".

  10. LOG() before AixLog::Log::init() (main.cpp)

  The "Another instance already running" error is logged before the logging system is initialized.

  11. Indentation issues in router.cpp and tray_app.cpp

  Extra indentation (8 spaces where 4 expected) introduced during the conversion.

  12. dynamic_mem_gb → virtual_mem_gb rename

  This is a data format breaking change. If any cached hardware JSON still uses "dynamic_mem_gb", the field will be silently ignored. Needs coordination.

  13. Removed default parameters (backend_utils.h, wrapped_server.h, fastflowlm_server.h)

  Default values removed from install_from_github(..., progress_cb = nullptr) and constructor log_level = "info". Any external callers relying on defaults will break at compile time.     

[superm1]

claude /review found some critical-level and moderate-level issues, thoughts?

Critical Issues

  1. ~19 instances of missing spaces in error messages (server.cpp)

  The migration introduced broken error strings like:
  "ERROR inhandle_completions"    // should be "ERROR in handle_completions"
  "JSON parse error inhandle_image_edits"
  This affects nearly every error handler in server.cpp. Clearly a find-and-replace artifact.

  2. #pragma once removed from single_instance.h

  The include guard was removed with no #ifndef replacement. This will cause multiple-definition errors if included from more than one translation unit.

  3. LOG() in signal handler (main.cpp)

  void signal_handler(int signal) {
      LOG(INFO) << "Shutdown signal received, exiting..." << std::endl;

  LOG() involves mutexes and memory allocation — not async-signal-safe. Use write(STDOUT_FILENO, ...) or keep the old std::cout (also technically unsafe but simpler).

  4. LOG() after fork() in child process (process_manager.cpp)

  After fork(), using LOG() (which holds mutexes cloned from the parent) is unsafe in multi-threaded programs.

  5. Doubled tags in log messages

  Several log messages now have redundant tags:
  LOG(ERROR, "FastFlowLM") << "[FastFlowLM ERROR] Failed to download..."
  LOG(DEBUG, "LlamaCpp") << "[LlamaCpp] Model loaded on port..."
  LOG(ERROR, "WrappedServer") << "[WrappedServer] Backend returned HTTP..."

  ---
  Moderate Issues

  6. Inconsistent timestamp format
  - main.cpp: "%Y-%m-%d %H:%M:%S.#ms" (colons)
  - tray/main.cpp: "%Y-%m-%d %H-%M-%S.#ms" (dashes)

  7. is_quiet_get — dead code (server.cpp)

  A variable is computed to detect low-signal GET requests but is never used. Looks like an incomplete implementation.

  8. Leftover std::cerr in fastflowlm_server.cpp

  Two std::cerr calls were not converted to LOG() right after a LOG(ERROR) call.

  9. Redundant "Warning:" prefix

  Several messages now read [WARNING] (ModelManager) Warning: Could not load... — double "warning".

  10. LOG() before AixLog::Log::init() (main.cpp)

  The "Another instance already running" error is logged before the logging system is initialized.

  11. Indentation issues in router.cpp and tray_app.cpp

  Extra indentation (8 spaces where 4 expected) introduced during the conversion.

  12. dynamic_mem_gb → virtual_mem_gb rename

  This is a data format breaking change. If any cached hardware JSON still uses "dynamic_mem_gb", the field will be silently ignored. Needs coordination.

  13. Removed default parameters (backend_utils.h, wrapped_server.h, fastflowlm_server.h)

  Default values removed from install_from_github(..., progress_cb = nullptr) and constructor log_level = "info". Any external callers relying on defaults will break at compile time.     

The data format I'm not worried about - we're going to axe cache on upgrade. A lot of the others make sense. Since you have it on /review already maybe just have it fix em and send it and we can review again.

[superm1]

claude /review found some critical-level and moderate-level issues, thoughts?

Critical Issues

  1. ~19 instances of missing spaces in error messages (server.cpp)

  The migration introduced broken error strings like:
  "ERROR inhandle_completions"    // should be "ERROR in handle_completions"
  "JSON parse error inhandle_image_edits"
  This affects nearly every error handler in server.cpp. Clearly a find-and-replace artifact.

  2. #pragma once removed from single_instance.h

  The include guard was removed with no #ifndef replacement. This will cause multiple-definition errors if included from more than one translation unit.

  3. LOG() in signal handler (main.cpp)

  void signal_handler(int signal) {
      LOG(INFO) << "Shutdown signal received, exiting..." << std::endl;

  LOG() involves mutexes and memory allocation — not async-signal-safe. Use write(STDOUT_FILENO, ...) or keep the old std::cout (also technically unsafe but simpler).

  4. LOG() after fork() in child process (process_manager.cpp)

  After fork(), using LOG() (which holds mutexes cloned from the parent) is unsafe in multi-threaded programs.

  5. Doubled tags in log messages

  Several log messages now have redundant tags:
  LOG(ERROR, "FastFlowLM") << "[FastFlowLM ERROR] Failed to download..."
  LOG(DEBUG, "LlamaCpp") << "[LlamaCpp] Model loaded on port..."
  LOG(ERROR, "WrappedServer") << "[WrappedServer] Backend returned HTTP..."

  ---
  Moderate Issues

  6. Inconsistent timestamp format
  - main.cpp: "%Y-%m-%d %H:%M:%S.#ms" (colons)
  - tray/main.cpp: "%Y-%m-%d %H-%M-%S.#ms" (dashes)

  7. is_quiet_get — dead code (server.cpp)

  A variable is computed to detect low-signal GET requests but is never used. Looks like an incomplete implementation.

  8. Leftover std::cerr in fastflowlm_server.cpp

  Two std::cerr calls were not converted to LOG() right after a LOG(ERROR) call.

  9. Redundant "Warning:" prefix

  Several messages now read [WARNING] (ModelManager) Warning: Could not load... — double "warning".

  10. LOG() before AixLog::Log::init() (main.cpp)

  The "Another instance already running" error is logged before the logging system is initialized.

  11. Indentation issues in router.cpp and tray_app.cpp

  Extra indentation (8 spaces where 4 expected) introduced during the conversion.

  12. dynamic_mem_gb → virtual_mem_gb rename

  This is a data format breaking change. If any cached hardware JSON still uses "dynamic_mem_gb", the field will be silently ignored. Needs coordination.

  13. Removed default parameters (backend_utils.h, wrapped_server.h, fastflowlm_server.h)

  Default values removed from install_from_github(..., progress_cb = nullptr) and constructor log_level = "info". Any external callers relying on defaults will break at compile time.     

The data format I'm not worried about - we're going to axe cache on upgrade. A lot of the others make sense. Since you have it on /review already maybe just have it fix em and send it and we can review again.

I've incorporated solutions for most of this and rebased.

[superm1]

@jeremyfowers Since this touches nearly all of the code base, can you review this again for merging soon? It needs a rebase hourly otherwise..

[jeremyfowers]

@jeremyfowers Since this touches nearly all of the code base, can you review this again for merging soon? It needs a rebase hourly otherwise..

Sorry, doing it now.