Page MenuHomeDevCentral

UTF-8 log entries aren't processed
Open, HighPublic

Description

When a log entry contains UTF-8 character, it's not published.

The suspects are our TCL code (write JSON) and JsonMapper (read JSON), but this last library can handle UTF-8 in rNOTIF code.

Bogus log entry

[Ysul] Manual port upgrade to nano 4.2 — "Tax the rich, pay the teachers"

Replacing the UTF-8 character by the ASCII character - works fine.

Client error stacktrace

18:29:04 <Wearg> [DEBUG] HTTP 500
18:29:04 <Wearg>     while executing
18:29:04 <Wearg> "::rest::_call {} $headers $url $query $body $error_body"
18:29:04 <Wearg>     (procedure "rest::simple" line 44)
18:29:04 <Wearg>     invoked from within
18:29:04 <Wearg> "rest::simple https://api.nasqueron.org/servers-log/ {} {
18:29:04 <Wearg>         method PUT
18:29:04 <Wearg>         content-type application/json
18:29:04 <Wearg>         format json
18:29:04 <Wearg>     } $request"
18:29:04 <Wearg>     (procedure "add_to_servers_log" line 10)
18:29:04 <Wearg>     invoked from within
18:29:04 <Wearg> "add_to_servers_log $emitter "$network $source" $component $entry"
18:29:04 <Wearg>     (procedure "handle_send_to_servers_log" line 6)
18:29:04 <Wearg>     invoked from within
18:29:04 <Wearg> "handle_send_to_servers_log [resolve_nick $nick] $chan $text $callback"
18:29:04 <Wearg>     (procedure "pubm:log" line 14)
18:29:04 <Wearg>     invoked from within
18:29:04 <Wearg> "pubm:log $_pubm1 $_pubm2 $_pubm3 $_pubm4 $_pubm5"

Server error stacktrace

PHP Fatal error:  Uncaught InvalidArgumentException: JsonMapper::map() requires first argument to be an object, NULL given. in /srv/api/servers-log/vendor/netresearch/jsonmapper/src/JsonMapper.php:126
Stack trace:
#0 /srv/api/servers-log/src/LogEntry.php(43): JsonMapper->map(NULL, Object(Nasqueron\Api\ServersLog\LogEntry))
#1 /srv/api/servers-log/src/Service.php(42): Nasqueron\Api\ServersLog\LogEntry::fromJSON(NULL)
#2 /srv/api/servers-log/src/Service.php(33): Nasqueron\Api\ServersLog\Service->put(NULL)
#3 /srv/api/servers-log/src/public/index.php(19): Nasqueron\Api\ServersLog\Service->handle()
#4 {main}
  thrown in /srv/api/servers-log/vendor/netresearch/jsonmapper/src/JsonMapper.php on line 126

Event Timeline

dereckson renamed this task from UTF-8 log entries aren't Malformed JSON is emitted by Wearg when quotes are used in log entry to UTF-8 log entries aren't processed.Apr 24 2019, 18:43
dereckson triaged this task as High priority.
dereckson created this task.
dereckson moved this task from Backlog to Bugs on the Dæghrefn board.

When the payload generated by rVIPER Wearg/ServersLog.tcl has been tested with Postman, it works fine.

The eggdrop environment is set like this:

$ env
06:37:23 <Wearg> LANG=en_US.UTF-8
06:37:23 <Wearg> MM_CHARSET=UTF-8
06:37:23 <Wearg> LC_ALL=en_US.UTF-8

A workaround could be to encode UTF-8 characters in the JSON payload, a choice rather popular in JSON libraries.

Note for lyrics, we're correctly writing in UTF-8 to the database, so this bug isn't related.