UTF-8 log entries aren't processed
Open, HighPublic
Actions

Assigned To

None

Authored By

	dereckson
	Apr 24 2019, 18:43

Description

When a log entry contains UTF-8 character, it's not published.

The suspects are our TCL code (write JSON) and JsonMapper (read JSON), but this last library can handle UTF-8 in rNOTIF code.

Bogus log entry

[Ysul] Manual port upgrade to nano 4.2 — "Tax the rich, pay the teachers"

Replacing the UTF-8 character — by the ASCII character - works fine.

Client error stacktrace

18:29:04 <Wearg> [DEBUG] HTTP 500
18:29:04 <Wearg>     while executing
18:29:04 <Wearg> "::rest::_call {} $headers $url $query $body $error_body"
18:29:04 <Wearg>     (procedure "rest::simple" line 44)
18:29:04 <Wearg>     invoked from within
18:29:04 <Wearg> "rest::simple https://api.nasqueron.org/servers-log/ {} {
18:29:04 <Wearg>         method PUT
18:29:04 <Wearg>         content-type application/json
18:29:04 <Wearg>         format json
18:29:04 <Wearg>     } $request"
18:29:04 <Wearg>     (procedure "add_to_servers_log" line 10)
18:29:04 <Wearg>     invoked from within
18:29:04 <Wearg> "add_to_servers_log $emitter "$network $source" $component $entry"
18:29:04 <Wearg>     (procedure "handle_send_to_servers_log" line 6)
18:29:04 <Wearg>     invoked from within
18:29:04 <Wearg> "handle_send_to_servers_log [resolve_nick $nick] $chan $text $callback"
18:29:04 <Wearg>     (procedure "pubm:log" line 14)
18:29:04 <Wearg>     invoked from within
18:29:04 <Wearg> "pubm:log $_pubm1 $_pubm2 $_pubm3 $_pubm4 $_pubm5"

Server error stacktrace

PHP Fatal error:  Uncaught InvalidArgumentException: JsonMapper::map() requires first argument to be an object, NULL given. in /srv/api/servers-log/vendor/netresearch/jsonmapper/src/JsonMapper.php:126
Stack trace:
#0 /srv/api/servers-log/src/LogEntry.php(43): JsonMapper->map(NULL, Object(Nasqueron\Api\ServersLog\LogEntry))
#1 /srv/api/servers-log/src/Service.php(42): Nasqueron\Api\ServersLog\LogEntry::fromJSON(NULL)
#2 /srv/api/servers-log/src/Service.php(33): Nasqueron\Api\ServersLog\Service->put(NULL)
#3 /srv/api/servers-log/src/public/index.php(19): Nasqueron\Api\ServersLog\Service->handle()
#4 {main}
  thrown in /srv/api/servers-log/vendor/netresearch/jsonmapper/src/JsonMapper.php on line 126

Event Timeline

dereckson renamed this task from UTF-8 log entries aren't Malformed JSON is emitted by Wearg when quotes are used in log entry to UTF-8 log entries aren't processed.Apr 24 2019, 18:43

dereckson triaged this task as High priority.

dereckson created this task.

dereckson moved this task from Backlog to Bugs on the Dæghrefn board.

When the payload generated by rVIPER Wearg/ServersLog.tcl has been tested with Postman, it works fine.

The eggdrop environment is set like this:

$ env
06:37:23 <Wearg> LANG=en_US.UTF-8
06:37:23 <Wearg> MM_CHARSET=UTF-8
06:37:23 <Wearg> LC_ALL=en_US.UTF-8

A workaround could be to encode UTF-8 characters in the JSON payload, a choice rather popular in JSON libraries.

dereckson moved this task from Backlog to Eggdrop dev on the IRC board.Mar 14 2022, 23:16

dereckson moved this task from Backlog to External / blocked / pending review on the Nasqueron API board.Dec 21 2022, 19:17

Note for lyrics, we're correctly writing in UTF-8 to the database, so this bug isn't related.

@yousra I've noticed I've already created a task with the same issue, with dash character — instead of the non-breaking space here.

It serves now a 400 error in that case, but the behavior bot-side is still the same: it converts the message with a UTF-8 character into a JSON message read as NULL by PHP code.

Recreating the JSON payload with dict2json [list date [iso8601date] emitter Dereckson source #nasqueron-ops component WindRiver entry {Manual port upgrade to nano 4.2 — "Tax the rich, pay the teachers"}], we've a correct JSON output, with the UTF-8 character encoded as is (see above message it should be escaped).

I suspect the PHP parser to ignore UTF-8 characters reading JSON message.

Code responsible to convert the message into a JSON payload, then post it:

Wearg/ServersLog.tcl

proc add_to_servers_log {emitter source component entry} {
    set request [dict2json "
        date [iso8601date]
        emitter $emitter
        source {$source}
        component {$component}
        entry {$entry}
    "]

    rest::simple https://api.nasqueron.org/servers-log/ {} {
        method PUT
        content-type application/json
        format json
    } $request
}

Core.tcl

proc dict2json {dictToEncode} {
    ::json::write object {*}[dict map {k v} $dictToEncode {
        set v [::json::write string $v]
    }]
}

UTF-8 log entries aren't processedOpen, HighPublicActions

Description

Event Timeline

UTF-8 log entries aren't processed
Open, HighPublic
Actions