python: send "bytes" instead of "str" to callbacks in Python 3 when the string is not UTF-8 valid (issue #1220, closes #1389)

ChangeLog.adoc

@@ -46,6 +46,7 @@ Bug fixes::
   * irc: remove option irc.network.channel_encode, add server option "charset_message" to control which part of the IRC message is decoded/encoded to the target charset (issue #832)
   * irc: use path from option xfer.file.upload_path to complete filename in command "/dcc send" (issue #60)
   * logger: fix write in log file if it has been deleted or renamed (issue #123)
+  * python: send "bytes" instead of "str" to callbacks in Python 3 when the string is not UTF-8 valid (issue #1389)
   * xfer: fix memory leak when a xfer is freed and when the plugin is unloaded
 
 Tests::

doc/de/weechat_scripting.de.adoc

@@ -3,9 +3,10 @@
 :email: flashcode@flashtux.org
 :lang: de
 :toc: left
-:toclevels: 3
+:toclevels: 4
 :toc-title: Inhaltsverzeichnis
 :sectnums:
+:sectnumlevels: 3
 :docinfo1:
 
 
@@ -73,22 +74,95 @@ und die Dokumentation für die Funktion `hook_process` in link:weechat_plugin_ap
 
 ==== Python
 
-* WeeChat muss als Modul eingebunden werden: `import weechat`
+// TRANSLATION MISSING
-* Um die WeeChat Funktion `+print*+` nutzen zu können muss `+prnt*+` genutzt
+===== Module
-  werden (_print_ ist ein reservierter Befehl von Python!)
+
-* Funktionen werden im Format `weechat.xxx(arg1, arg2, ...)` ausgeführt
+WeeChat defines a `weechat` module which must be imported with `import weechat`.
+
+// TRANSLATION MISSING
+===== Functions
+
+Functions are called with `weechat.xxx(arg1, arg2, ...)`.
+
+Functions `+print*+` are called `+prnt*+` in python (because `print` was a
+reserved keyword in Python 2).
+
+// TRANSLATION MISSING
+===== Strings received in callbacks
+
+In Python 3 and with WeeChat ≥ 2.7, the strings received in callbacks have type
+`str` if the string has valid UTF-8 data (which is the most common case),
+or `bytes` if the string is not UTF-8 valid. So the callback should take care
+about this type if some invalid UTF-8 content can be received.
+
+Some invalid UTF-8 data may be received in these cases, so the callback can
+receive a string of type `str` or `bytes` (this list is not exhaustive):
+
+[width="100%",cols="3m,3m,3m,8",options="header"]
+|===
+| API function | Arguments | Examples | Description
+
+| hook_modifier |
+  irc_in_yyy |
+  pass:[irc_in_privmsg] +
+  pass:[irc_in_notice] |
+  A message received in IRC plugin, before it is decoded to UTF-8 (used
+  internally). +
+  +
+  It is recommended to use modifier `irc_in2_yyy` instead, the string received
+  is always UTF-8 valid. +
+  See function `hook_modifier` in the
+  link:weechat_plugin_api.en.html#_hook_modifier[WeeChat plugin API reference].
+
+| hook_signal |
+  xxx,irc_out_yyy +
+  xxx,irc_outtags_yyy |
+  pass:[*,irc_out_privmsg] +
+  pass:[*,irc_out_notice] +
+  pass:[*,irc_outtags_privmsg] +
+  pass:[*,irc_outtags_notice] |
+  A message sent by IRC plugin, after it is encoded to the `encode` charset
+  defined by the user (if different from the default `UTF-8`). +
+  +
+  It is recommended to use signal `xxx,irc_out1_yyy` instead, the string received
+  is always UTF-8 valid. +
+  See function `hook_signal` in the
+  link:weechat_plugin_api.en.html#_hook_signal[WeeChat plugin API reference].
+
+| hook_process +
+  hook_process_hashtable |
+  - |
+  - |
+  Output of the command, sent to the callback, can contain invalid UTF-8 data.
+
+|===
+
+In Python 2, which is now deprecated and should not be used any more, the
+strings sent to callbacks were always of type `str`, and may contain invalid
+UTF-8 data, in the cases mentioned above.
 
 ==== Perl
 
-* Funktionen werden im Format `weechat::xxx(arg1, arg2, ...);` ausgeführt
+// TRANSLATION MISSING
+===== Functions
+
+Functions are called with `weechat::xxx(arg1, arg2, ...);`.
 
 ==== Ruby
 
-* Es muss _weechat_init_ definiert und darin die Funktion _register_ ausgeführt werden
+// TRANSLATION MISSING
-* Funktionen werden im Format `Weechat.xxx(arg1, arg2, ...)` ausgeführt
+===== Initialization
-* Aufgrund einer Limitierung, seitens Ruby (maximal 15 Argumente pro Funktion), empfängt
+
-  die Funktion `Weechat.config_new_option` den Callback in einem Array von 6 Strings
+You have to define _weechat_init_ and call _register_ inside.
-  (3 Callbacks + 3 Data Strings), somit sieht ein Aufruf der Funktion folgendermaßen aus:
+
+// TRANSLATION MISSING
+===== Functions
+
+Functions are called with `Weechat.xxx(arg1, arg2, ...)`.
+
+Due to a limitation of Ruby (15 arguments max by function), the function
+`Weechat.config_new_option` receives the callbacks in an array of 6 strings
+(3 callbacks + 3 data strings), so a call to this function looks like:
 
 [source,ruby]
 ----
@@ -98,29 +172,46 @@ Weechat.config_new_option(config, section, "name", "string", "description of opt
 
 ==== Lua
 
-* Funktionen werden im Format `weechat.xxx(arg1, arg2, ...)` ausgeführt
+// TRANSLATION MISSING
+===== Functions
+
+Functions are called with `weechat.xxx(arg1, arg2, ...)`.
 
 ==== Tcl
 
-* Funktionen werden im Format `weechat::xxx arg1 arg2 ...` ausgeführt
+// TRANSLATION MISSING
+===== Functions
+
+Functions are called with `weechat::xxx arg1 arg2 ...`.
 
 ==== Guile (Scheme)
 
-* Funktionen werden im Format `(weechat:xxx arg1 arg2 ...)` ausgeführt
+// TRANSLATION MISSING
-* folgende Funktionen nutzen eine Liste von Argumente (anstelle von vielen
+===== Functions
-  Argumenten für andere Funktionen), dies liegt daran das Guile die Anzahl
+
-  der Argumente eingeschränkt ist:
+Functions are called with `(weechat:xxx arg1 arg2 ...)`.
-** config_new_section
+
-** config_new_option
+The following functions take one list of arguments (instead of many arguments
-** bar_new
+for other functions), because number of arguments exceed number of allowed
+arguments in Guile:
+
+* config_new_section
+* config_new_option
+* bar_new
 
 ==== JavaScript
 
-* Funktionen werden im Format `weechat.xxx(arg1, arg2, ...);` ausgeführt
+// TRANSLATION MISSING
+===== Functions
+
+Functions are called with `weechat.xxx(arg1, arg2, ...);`.
 
 ==== PHP
 
-* Funktionen werden im Format `weechat_xxx(arg1, arg2, ...);` ausgeführt
+// TRANSLATION MISSING
+===== Functions
+
+Functions are called with `weechat_xxx(arg1, arg2, ...);`.
 
 [[register_function]]
 === Die "Register" Funktion
@@ -1103,15 +1194,25 @@ weechat.prnt("", "Wert der Option weechat.color.chat_delimiters ist: %s"
 [[irc_catch_messages]]
 ==== Nachrichten abfangen
 
-Die IRC Erweiterung sendet zwei Signale wenn eine Nachricht empfangen wurde.
+// TRANSLATION MISSING
-`xxx` ist der interne IRC Servername, `yyy` ist der IRC Befehl der empfangen
+IRC plugin sends four signals for a message received (`xxx` is IRC internal
-wurde (JOIN, QUIT, PRIVMSG, 301, ..):
+server name, `yyy` is IRC command name like JOIN, QUIT, PRIVMSG, 301, ..):
 
-xxxx,irc_in_yyy::
+// TRANSLATION MISSING
-    Signal wird gesendet bevor die Nachricht verarbeitet wurde.
+xxx,irc_in_yyy::
+    signal sent before processing message, only if message is *not* ignored
 
+// TRANSLATION MISSING
 xxx,irc_in2_yyy::
-    Signal wird gesendet nachdem die Nachricht verarbeitet wurde.
+    signal sent after processing message, only if message is *not* ignored
+
+// TRANSLATION MISSING
+xxx,irc_raw_in_yyy::
+    signal sent before processing message, even if message is ignored
+
+// TRANSLATION MISSING
+xxx,irc_raw_in2_yyy::
+    signal sent after processing message, even if message is ignored
 
 [source,python]
 ----
@@ -1133,8 +1234,19 @@ weechat.hook_signal("*,irc_in2_join", "join_cb", "")
 [[irc_modify_messages]]
 ==== Nachrichten ändern
 
-Die IRC Erweiterung verschickt einen "modifier" mit Namen "irc_in_xxx" ("xxx" steht für den
+// TRANSLATION MISSING
-Namen des IRC Befehls) falls eine Nachricht empfangen wurde die dann modifiziert werden kann.
+IRC plugin sends two "modifiers" for a message received ("xxx" is IRC command),
+so that you can modify it:
+
+// TRANSLATION MISSING
+irc_in_xxx::
+    modifier sent before charset decoding: use with caution, the string may
+    contain invalid UTF-8 data; use only for raw operations on a message
+
+// TRANSLATION MISSING
+irc_in2_xxx::
+    modifier sent after charset decoding, so the string received is always
+    UTF-8 valid (*recommended*)
 
 [source,python]
 ----
@@ -1143,7 +1255,7 @@ def modifier_cb(data, modifier, modifier_data, string):
     # (Okay dies ist nicht wirklich sinnvoll, aber es ist auch nur ein Beispiel!)
     return "%s %s" % (string, modifier_data)
 
-weechat.hook_modifier("irc_in_privmsg", "modifier_cb", "")
+weechat.hook_modifier("irc_in2_privmsg", "modifier_cb", "")
 ----
 
 [WARNING]

doc/en/weechat_plugin_api.en.adoc

@@ -9871,12 +9871,16 @@ List of signals sent by WeeChat and plugins:
 | irc | xxx,irc_out_yyy ^(1)^ |
   String: message. |
   IRC message sent to server after automatic split
-  (to fit in 512 bytes by default).
+  (to fit in 512 bytes by default). +
+  *Warning:* the string may contain invalid UTF-8 data.
+  Signal "xxx,irc_out1_yyy" is recommended instead.
 
 | irc | xxx,irc_outtags_yyy ^(1)^ +
   _(WeeChat ≥ 0.3.4)_ |
   String: tags + ";" + message. |
-  Tags + IRC message sent to server.
+  Tags + IRC message sent to server. +
+  *Warning:* the string may contain invalid UTF-8 data.
+  Signal "xxx,irc_out1_yyy" is recommended instead.
 
 | irc | irc_ctcp |
   String: message. |
@@ -11214,7 +11218,10 @@ List of modifiers used by WeeChat and plugins:
 
 | [[hook_modifier_irc_in_xxx]] irc_in_xxx ^(1)^ |
   Server name |
-  Content of message received from IRC server (before charset decoding). |
+  Content of message received from IRC server (before charset decoding). +
+  *Warning:* the string may contain invalid UTF-8 data; use only for raw
+  operations on a message.
+  Modifier <<hook_modifier_irc_in2_xxx,irc_in2_xxx>> is recommended instead. |
   New content of message.
 
 | [[hook_modifier_irc_in2_xxx]] irc_in2_xxx ^(1)^ +

doc/en/weechat_scripting.en.adoc

@@ -3,8 +3,9 @@
 :email: flashcode@flashtux.org
 :lang: en
 :toc: left
-:toclevels: 3
+:toclevels: 4
 :sectnums:
+:sectnumlevels: 3
 :docinfo1:
 
 
@@ -67,22 +68,89 @@ link:weechat_plugin_api.en.html#_hook_process[WeeChat plugin API reference].
 
 ==== Python
 
-* You have to `import weechat`.
+===== Module
-* Functions `+print*+` are called `+prnt*+` in python (because _print_ is reserved
+
-  keyword).
+WeeChat defines a `weechat` module which must be imported with `import weechat`.
-* Functions are called with `weechat.xxx(arg1, arg2, ...)`.
+
+===== Functions
+
+Functions are called with `weechat.xxx(arg1, arg2, ...)`.
+
+Functions `+print*+` are called `+prnt*+` in python (because `print` was a
+reserved keyword in Python 2).
+
+===== Strings received in callbacks
+
+In Python 3 and with WeeChat ≥ 2.7, the strings received in callbacks have type
+`str` if the string has valid UTF-8 data (which is the most common case),
+or `bytes` if the string is not UTF-8 valid. So the callback should take care
+about this type if some invalid UTF-8 content can be received.
+
+Some invalid UTF-8 data may be received in these cases, so the callback can
+receive a string of type `str` or `bytes` (this list is not exhaustive):
+
+[width="100%",cols="3m,3m,3m,8",options="header"]
+|===
+| API function | Arguments | Examples | Description
+
+| hook_modifier |
+  irc_in_yyy |
+  pass:[irc_in_privmsg] +
+  pass:[irc_in_notice] |
+  A message received in IRC plugin, before it is decoded to UTF-8 (used
+  internally). +
+  +
+  It is recommended to use modifier `irc_in2_yyy` instead, the string received
+  is always UTF-8 valid. +
+  See function `hook_modifier` in the
+  link:weechat_plugin_api.en.html#_hook_modifier[WeeChat plugin API reference].
+
+| hook_signal |
+  xxx,irc_out_yyy +
+  xxx,irc_outtags_yyy |
+  pass:[*,irc_out_privmsg] +
+  pass:[*,irc_out_notice] +
+  pass:[*,irc_outtags_privmsg] +
+  pass:[*,irc_outtags_notice] |
+  A message sent by IRC plugin, after it is encoded to the `encode` charset
+  defined by the user (if different from the default `UTF-8`). +
+  +
+  It is recommended to use signal `xxx,irc_out1_yyy` instead, the string received
+  is always UTF-8 valid. +
+  See function `hook_signal` in the
+  link:weechat_plugin_api.en.html#_hook_signal[WeeChat plugin API reference].
+
+| hook_process +
+  hook_process_hashtable |
+  - |
+  - |
+  Output of the command, sent to the callback, can contain invalid UTF-8 data.
+
+|===
+
+In Python 2, which is now deprecated and should not be used any more, the
+strings sent to callbacks were always of type `str`, and may contain invalid
+UTF-8 data, in the cases mentioned above.
 
 ==== Perl
 
-* Functions are called with `weechat::xxx(arg1, arg2, ...);`.
+===== Functions
+
+Functions are called with `weechat::xxx(arg1, arg2, ...);`.
 
 ==== Ruby
 
-* You have to define _weechat_init_ and call _register_ inside.
+===== Initialization
-* Functions are called with `Weechat.xxx(arg1, arg2, ...)`.
+
-* Due to a limitation of Ruby (15 arguments max by function), the function
+You have to define _weechat_init_ and call _register_ inside.
-  `Weechat.config_new_option` receives the callbacks in an array of 6 strings
+
-  (3 callbacks + 3 data strings), so a call to this function looks like:
+===== Functions
+
+Functions are called with `Weechat.xxx(arg1, arg2, ...)`.
+
+Due to a limitation of Ruby (15 arguments max by function), the function
+`Weechat.config_new_option` receives the callbacks in an array of 6 strings
+(3 callbacks + 3 data strings), so a call to this function looks like:
 
 [source,ruby]
 ----
@@ -92,29 +160,41 @@ Weechat.config_new_option(config, section, "name", "string", "description of opt
 
 ==== Lua
 
-* Functions are called with `weechat.xxx(arg1, arg2, ...)`.
+===== Functions
+
+Functions are called with `weechat.xxx(arg1, arg2, ...)`.
 
 ==== Tcl
 
-* Functions are called with `weechat::xxx arg1 arg2 ...`.
+===== Functions
+
+Functions are called with `weechat::xxx arg1 arg2 ...`.
 
 ==== Guile (Scheme)
 
-* Functions are called with `(weechat:xxx arg1 arg2 ...)`.
+===== Functions
-* Following functions take one list of arguments (instead of many arguments
+
-  for other functions), because number of arguments exceed number of allowed
+Functions are called with `(weechat:xxx arg1 arg2 ...)`.
-  arguments in Guile:
+
-** config_new_section
+The following functions take one list of arguments (instead of many arguments
-** config_new_option
+for other functions), because number of arguments exceed number of allowed
-** bar_new
+arguments in Guile:
+
+* config_new_section
+* config_new_option
+* bar_new
 
 ==== JavaScript
 
-* Functions are called with `weechat.xxx(arg1, arg2, ...);`.
+===== Functions
+
+Functions are called with `weechat.xxx(arg1, arg2, ...);`.
 
 ==== PHP
 
-* Functions are called with `weechat_xxx(arg1, arg2, ...);`.
+===== Functions
+
+Functions are called with `weechat_xxx(arg1, arg2, ...);`.
 
 [[register_function]]
 === Register function
@@ -1081,14 +1161,20 @@ weechat.prnt("", "value of option weechat.color.chat_delimiters is: %s"
 [[irc_catch_messages]]
 ==== Catch messages
 
-IRC plugin sends two signals for a message received (`xxx` is IRC internal
+IRC plugin sends four signals for a message received (`xxx` is IRC internal
 server name, `yyy` is IRC command name like JOIN, QUIT, PRIVMSG, 301, ..):
 
-xxxx,irc_in_yyy::
+xxx,irc_in_yyy::
-    signal sent before processing message
+    signal sent before processing message, only if message is *not* ignored
 
 xxx,irc_in2_yyy::
-    signal sent after processing message
+    signal sent after processing message, only if message is *not* ignored
+
+xxx,irc_raw_in_yyy::
+    signal sent before processing message, even if message is ignored
+
+xxx,irc_raw_in2_yyy::
+    signal sent after processing message, even if message is ignored
 
 [source,python]
 ----
@@ -1110,8 +1196,16 @@ weechat.hook_signal("*,irc_in2_join", "join_cb", "")
 [[irc_modify_messages]]
 ==== Modify messages
 
-IRC plugin sends a "modifier" called "irc_in_xxx" ("xxx" is IRC command) for a
+IRC plugin sends two "modifiers" for a message received ("xxx" is IRC command),
-message received, so that you can modify it.
+so that you can modify it:
+
+irc_in_xxx::
+    modifier sent before charset decoding: use with caution, the string may
+    contain invalid UTF-8 data; use only for raw operations on a message
+
+irc_in2_xxx::
+    modifier sent after charset decoding, so the string received is always
+    UTF-8 valid (*recommended*)
 
 [source,python]
 ----
@@ -1120,7 +1214,7 @@ def modifier_cb(data, modifier, modifier_data, string):
     # (OK that's not very useful, but that's just an example!)
     return "%s %s" % (string, modifier_data)
 
-weechat.hook_modifier("irc_in_privmsg", "modifier_cb", "")
+weechat.hook_modifier("irc_in2_privmsg", "modifier_cb", "")
 ----
 
 [WARNING]

doc/fr/weechat_plugin_api.fr.adoc

@@ -10072,12 +10072,16 @@ Liste des signaux envoyés par WeeChat et les extensions :
   _(WeeChat ≥ 0.3.7)_ |
   Chaîne : message. |
   Message IRC envoyé au serveur avant découpage automatique
-  (pour tenir dans les 512 octets par défaut).
+  (pour tenir dans les 512 octets par défaut). +
+  *Attention :* la chaîne peut contenir des données invalides UTF-8.
+  Le signal "xxx,irc_out1_yyy" est recommandé à la place de celui-ci.
 
 | irc | xxx,irc_out_yyy ^(1)^ |
   Chaîne : message. |
   Message IRC envoyé au serveur après découpage automatique
-  (pour tenir dans les 512 octets par défaut).
+  (pour tenir dans les 512 octets par défaut). +
+  *Attention :* la chaîne peut contenir des données invalides UTF-8.
+  Le signal "xxx,irc_out1_yyy" est recommandé à la place de celui-ci.
 
 | irc | xxx,irc_outtags_yyy ^(1)^ +
   _(WeeChat ≥ 0.3.4)_ |
@@ -11449,7 +11453,11 @@ Liste des modificateurs utilisés par WeeChat et les extensions :
 
 | [[hook_modifier_irc_in_xxx]] irc_in_xxx ^(1)^ |
   Nom de serveur |
-  Contenu du message reçu du serveur IRC (avant décodage du jeu de caractères). |
+  Contenu du message reçu du serveur IRC (avant décodage du jeu de caractères). +
+  *Attention :* la chaîne peut contenir des données invalides UTF-8 ; à utiliser
+  seulement pour les opérations de bas niveau sur le message.
+  Le modificateur <<hook_modifier_irc_in2_xxx,irc_in2_xxx>> est recommandé à la
+  place de celui-ci. |
   Nouveau contenu du message.
 
 | [[hook_modifier_irc_in2_xxx]] irc_in2_xxx ^(1)^ +

doc/fr/weechat_scripting.fr.adoc

@@ -3,9 +3,10 @@
 :email: flashcode@flashtux.org
 :lang: fr
 :toc: left
-:toclevels: 3
+:toclevels: 4
 :toc-title: Table des matières
 :sectnums:
+:sectnumlevels: 3
 :docinfo1:
 
 
@@ -73,23 +74,93 @@ link:weechat_plugin_api.en.html#_hook_process[Référence API extension WeeChat]
 
 ==== Python
 
-* Vous devez utiliser `import weechat`.
+===== Module
-* Les fonctions `+print*+` se nomment `+prnt*+` en python (car _print_ est un mot
+
-  clé réservé).
+WeeChat définit un module `weechat` qui doit être importé avec `import weechat`.
-* Les fonctions sont appelées par `weechat.xxx(arg1, arg2, ...)`.
+
+===== Fonctions
+
+Les fonctions sont appelées avec `weechat.xxx(arg1, arg2, ...)`.
+
+Les fonctions `+print*+` se nomment `+prnt*+` en python (car `print` était un
+mot clé réservé en Python 2).
+
+===== Chaînes reçues dans les fonctions de rappel
+
+En Python 3 et avec WeeChat ≥ 2.7, les chaînes reçues dans les fonctions de
+rappel ont le type `str` si la chaîne a des données valides UTF-8 (ce qui est
+le cas le plus courant) ou `bytes` si la chaîne n'est pas valide UTF-8. Donc la
+fonction de rappel doit prendre en compte ce type si des données non valides
+UTF-8 peuvent être reçues.
+
+Des données invalides UTF-8 peuvent être reçues dans ces cas, donc la fonction
+de rappel peut recevoir une chaîne de type `str` ou `bytes` (cette liste n'est
+pas exhaustive) :
+
+[width="100%",cols="3m,3m,3m,8",options="header"]
+|===
+| Fonction API | Paramètres | Exemples | Description
+
+| hook_modifier |
+  irc_in_yyy |
+  pass:[irc_in_privmsg] +
+  pass:[irc_in_notice] |
+  Un message reçu dans l'extension IRC, avant qu'il ne soit décodé vers UTF-8. +
+  +
+  Il est recommandé d'utiliser plutôt le modificateur `irc_in2_yyy`, la chaîne
+  reçue sera toujours valide UTF-8. +
+  Voir la fonction `hook_modifier` dans la
+  link:weechat_plugin_api.fr.html#_hook_modifier[Référence API extension WeeChat].
+
+| hook_signal |
+  xxx,irc_out_yyy +
+  xxx,irc_outtags_yyy |
+  pass:[*,irc_out_privmsg] +
+  pass:[*,irc_out_notice] +
+  pass:[*,irc_outtags_privmsg] +
+  pass:[*,irc_outtags_notice] |
+  Un message envoyé par l'extension IRC, après encodage vers le jeu de caractères
+  `encode` défini par l'utilisateur (si différent de `UTF-8`, qui est la valeur
+  par défaut). +
+  +
+  Il est recommandé d'utiliser plutôt le signal `xxx,irc_out1_yyy`, la chaîne
+  reçue sera toujours valide UTF-8. +
+  Voir la fonction `hook_signal` dans la
+  link:weechat_plugin_api.fr.html#_hook_signal[Référence API extension WeeChat].
+
+| hook_process +
+  hook_process_hashtable |
+  - |
+  - |
+  La sortie de la commande, envoyée à la fonction de rappel, peut contenir des
+  données invalides UTF-8.
+
+|===
+
+En Python 2, qui est déconseillé et ne devrait plus être utilisé, les chaînes
+envoyées aux fonctions de rappel sont toujours de type `str`, et peuvent contenir
+des données invalides UTF-8, dans les cas mentionnés ci-dessus.
 
 ==== Perl
 
-* Les fonctions sont appelées par `weechat::xxx(arg1, arg2, ...);`.
+===== Fonctions
+
+Les fonctions sont appelées par `weechat::xxx(arg1, arg2, ...);`.
 
 ==== Ruby
 
-* Vous devez définir _weechat_init_ et appeler _register_ dedans.
+===== Initialisation
-* Les fonctions sont appelées par `Weechat.xxx(arg1, arg2, ...)`.
+
-* En raison d'une limitation de Ruby (15 paramètres maximum par fonction), la
+Vous devez définir _weechat_init_ et appeler _register_ dedans.
-  fonction `Weechat.config_new_option` reçoit les fonctions de rappel dans un tableau de
+
-  6 chaînes de caractères (3 fonctions de rappel + 3 chaînes de données), donc un appel à
+===== Fonctions
-  cette fonction ressemble à ceci :
+
+Les fonctions sont appelées par `Weechat.xxx(arg1, arg2, ...)`.
+
+En raison d'une limitation de Ruby (15 paramètres maximum par fonction), la
+fonction `Weechat.config_new_option` reçoit les fonctions de rappel dans un
+tableau de 6 chaînes de caractères (3 fonctions de rappel + 3 chaînes de
+données), donc un appel à cette fonction ressemble à ceci :
 
 [source,ruby]
 ----
@@ -99,29 +170,41 @@ Weechat.config_new_option(config, section, "name", "string", "description of opt
 
 ==== Lua
 
-* Les fonctions sont appelées par `weechat.xxx(arg1, arg2, ...)`.
+===== Fonctions
+
+Les fonctions sont appelées par `weechat.xxx(arg1, arg2, ...)`.
 
 ==== Tcl
 
-* Les fonctions sont appelées par `weechat::xxx arg1 arg2 ...`.
+===== Fonctions
+
+Les fonctions sont appelées par `weechat::xxx arg1 arg2 ...`.
 
 ==== Guile (Scheme)
 
-* Les fonctions sont appelées par `(weechat:xxx arg1 arg2 ...)`.
+===== Fonctions
-* Les fonctions suivantes prennent une liste de paramètres en entrée (au lieu
+
-  de plusieurs paramètres pour les autres fonctions), car le nombre de
+Les fonctions sont appelées par `(weechat:xxx arg1 arg2 ...)`.
-  paramètres excède la limite de Guile :
+
-** config_new_section
+Les fonctions suivantes prennent une liste de paramètres en entrée (au lieu de
-** config_new_option
+plusieurs paramètres pour les autres fonctions), car le nombre de paramètres
-** bar_new
+excède la limite de Guile :
+
+* config_new_section
+* config_new_option
+* bar_new
 
 ==== JavaScript
 
-* Les fonctions sont appelées par `weechat.xxx(arg1, arg2, ...);`.
+===== Fonctions
+
+Les fonctions sont appelées par `weechat.xxx(arg1, arg2, ...);`.
 
 ==== PHP
 
-* Les fonctions sont appelées par `weechat_xxx(arg1, arg2, ...);`.
+===== Fonctions
+
+Les fonctions sont appelées par `weechat_xxx(arg1, arg2, ...);`.
 
 [[register_function]]
 === Fonction register
@@ -1109,15 +1192,23 @@ weechat.prnt("", "la valeur de l'option weechat.color.chat_delimiters est : %s"
 [[irc_catch_messages]]
 ==== Intercepter des messages
 
-L'extension IRC envoie deux signaux pour un message reçu (`xxx` est le nom
+L'extension IRC envoie quatre signaux pour un message reçu (`xxx` est le nom
 interne du serveur IRC, `yyy` est le nom de la commande IRC comme JOIN, QUIT,
 PRIVMSG, 301, ..) :
 
-xxxx,irc_in_yyy::
+xxx,irc_in_yyy::
-    signal envoyé avant traitement du message
+    signal envoyé avant traitement du message, uniquement si le message n'est
+    *pas* ignoré
 
 xxx,irc_in2_yyy::
-    message sent après traitement du message
+    signal envoyé après traitement du message, uniquement si le message n'est
+    *pas* ignoré
+
+xxx,irc_raw_in_yyy::
+    signal envoyé avant traitement du message, même si le message est ignoré
+
+xxx,irc_raw_in2_yyy::
+    signal envoyé après traitement du message, même si le message est ignoré
 
 [source,python]
 ----
@@ -1139,8 +1230,17 @@ weechat.hook_signal("*,irc_in2_join", "join_cb", "")
 [[irc_modify_messages]]
 ==== Modifier des messages
 
-L'extension IRC envoie un modificateur appelé "irc_in_xxx" ("xxx" est la
+L'extension IRC envoie deux modificateurs pour un message reçu ("xxx" est la
-commande IRC) pour un message reçu, de sorte que vous puissiez le modifier.
+commande IRC), de sorte que vous puissiez le modifier :
+
+irc_in_xxx::
+    modificateur envoyé avant le décodage du jeu de caractères : à utiliser avec
+    précaution, la chaîne peut contenir des données invalides UTF-8 ; à utiliser
+    seulement pour les opérations de bas niveau sur le message
+
+irc_in2_xxx::
+    modificateur envoyé après décodage du jeu de caractères, donc la chaîne
+    reçue est toujours valide UTF-8 (*recommendé*)
 
 [source,python]
 ----
@@ -1149,7 +1249,7 @@ def modifier_cb(data, modifier, modifier_data, string):
     # (ok ce n'est pas très utile, mais c'est juste un exemple !)
     return "%s %s" % (string, modifier_data)
 
-weechat.hook_modifier("irc_in_privmsg", "modifier_cb", "")
+weechat.hook_modifier("irc_in2_privmsg", "modifier_cb", "")
 ----
 
 [WARNING]

doc/it/weechat_plugin_api.it.adoc

@@ -10209,12 +10209,17 @@ List of signals sent by WeeChat and plugins:
 | irc | xxx,irc_out_yyy ^(1)^ |
   String: messaggio. |
   IRC message sent to server after automatic split
-  (to fit in 512 bytes by default).
+  (to fit in 512 bytes by default). +
+  *Warning:* the string may contain invalid UTF-8 data.
+  Signal "xxx,irc_out1_yyy" is recommended instead.
 
+// TRANSLATION MISSING
 | irc | xxx,irc_outtags_yyy ^(1)^ +
   _(WeeChat ≥ 0.3.4)_ |
   Stringa: tag + ";" + messaggio. |
-  Tag + messaggio IRC inviato al server.
+  Tag + messaggio IRC inviato al server. +
+  *Warning:* the string may contain invalid UTF-8 data.
+  Signal "xxx,irc_out1_yyy" is recommended instead.
 
 | irc | irc_ctcp |
   String: messaggio. |
@@ -11663,9 +11668,13 @@ List of modifiers used by WeeChat and plugins:
 |===
 | Modificatore | Dati modificatore | Stringa | Output
 
+// TRANSLATION MISSING
 | [[hook_modifier_irc_in_xxx]] irc_in_xxx ^(1)^ |
   Nome server |
-  Contenuto del messaggio ricevuto dal server IRC (prima della codifica del set caratteri). |
+  Contenuto del messaggio ricevuto dal server IRC (prima della codifica del set caratteri). +
+  *Warning:* the string may contain invalid UTF-8 data; use only for raw
+  operations on a message.
+  Modifier <<hook_modifier_irc_in2_xxx,irc_in2_xxx>> is recommended instead. |
   Nuovo contenuto del messaggio.
 
 | [[hook_modifier_irc_in2_xxx]] irc_in2_xxx ^(1)^ +

doc/it/weechat_scripting.it.adoc

@@ -3,9 +3,10 @@
 :email: flashcode@flashtux.org
 :lang: it
 :toc: left
-:toclevels: 3
+:toclevels: 4
 :toc-title: Indice
 :sectnums:
+:sectnumlevels: 3
 :docinfo1:
 
 
@@ -76,23 +77,95 @@ link:weechat_plugin_api.it.html#_hook_process[WeeChat plugin API reference].
 
 ==== Python
 
-* E necessario `import weechat`
+// TRANSLATION MISSING
-* Le funzioni `+print*+` sono chiamate `+prnt*+` in python (dato che _print_
+===== Module
-  è una parola riservata)
+
-* Le funzioni sono chiamate con `weechat.xxx(arg1, arg2, ...)`
+WeeChat defines a `weechat` module which must be imported with `import weechat`.
+
+// TRANSLATION MISSING
+===== Functions
+
+Functions are called with `weechat.xxx(arg1, arg2, ...)`.
+
+Functions `+print*+` are called `+prnt*+` in python (because `print` was a
+reserved keyword in Python 2).
+
+// TRANSLATION MISSING
+===== Strings received in callbacks
+
+In Python 3 and with WeeChat ≥ 2.7, the strings received in callbacks have type
+`str` if the string has valid UTF-8 data (which is the most common case),
+or `bytes` if the string is not UTF-8 valid. So the callback should take care
+about this type if some invalid UTF-8 content can be received.
+
+Some invalid UTF-8 data may be received in these cases, so the callback can
+receive a string of type `str` or `bytes` (this list is not exhaustive):
+
+[width="100%",cols="3m,3m,3m,8",options="header"]
+|===
+| API function | Arguments | Examples | Description
+
+| hook_modifier |
+  irc_in_yyy |
+  pass:[irc_in_privmsg] +
+  pass:[irc_in_notice] |
+  A message received in IRC plugin, before it is decoded to UTF-8 (used
+  internally). +
+  +
+  It is recommended to use modifier `irc_in2_yyy` instead, the string received
+  is always UTF-8 valid. +
+  See function `hook_modifier` in the
+  link:weechat_plugin_api.en.html#_hook_modifier[WeeChat plugin API reference].
+
+| hook_signal |
+  xxx,irc_out_yyy +
+  xxx,irc_outtags_yyy |
+  pass:[*,irc_out_privmsg] +
+  pass:[*,irc_out_notice] +
+  pass:[*,irc_outtags_privmsg] +
+  pass:[*,irc_outtags_notice] |
+  A message sent by IRC plugin, after it is encoded to the `encode` charset
+  defined by the user (if different from the default `UTF-8`). +
+  +
+  It is recommended to use signal `xxx,irc_out1_yyy` instead, the string received
+  is always UTF-8 valid. +
+  See function `hook_signal` in the
+  link:weechat_plugin_api.en.html#_hook_signal[WeeChat plugin API reference].
+
+| hook_process +
+  hook_process_hashtable |
+  - |
+  - |
+  Output of the command, sent to the callback, can contain invalid UTF-8 data.
+
+|===
+
+In Python 2, which is now deprecated and should not be used any more, the
+strings sent to callbacks were always of type `str`, and may contain invalid
+UTF-8 data, in the cases mentioned above.
 
 ==== Perl
 
-* Le funzioni sono chiamate con `weechat::xxx(arg1, arg2, ...);`
+// TRANSLATION MISSING
+===== Functions
+
+Functions are called with `weechat::xxx(arg1, arg2, ...);`.
 
 ==== Ruby
 
-* E necessario definire _weechat_init_ e chiamare _register_ all'interno
+// TRANSLATION MISSING
-* Le funzioni sono chiamate con `Weechat.xxx(arg1, arg2, ...)`
+===== Initialization
-* A causa di una limitazione di Ruby (massimo 15 argomenti per funzione), la
+
-  funzione `WeeChat.config_new_option` riceve le callback in un array di 6
+You have to define _weechat_init_ and call _register_ inside.
-  stringhe (3 callback + 3 stringhe di dati), così che una chiamata a questa
+
-  funzione appare come:
+// TRANSLATION MISSING
+===== Functions
+
+Functions are called with `Weechat.xxx(arg1, arg2, ...)`.
+
+Due to a limitation of Ruby (15 arguments max by function), the function
+`Weechat.config_new_option` receives the callbacks in an array of 6 strings
+(3 callbacks + 3 data strings), so a call to this function looks like:
 
 [source,ruby]
 ----
@@ -102,29 +175,46 @@ Weechat.config_new_option(config, section, "name", "string", "description of opt
 
 ==== Lua
 
-* Le funzioni sono chiamate con `weechat.xxx(arg1, arg2, ...)`
+// TRANSLATION MISSING
+===== Functions
+
+Functions are called with `weechat.xxx(arg1, arg2, ...)`.
 
 ==== Tcl
 
-* Le funzioni sono chiamate con `weechat::xxx arg1 arg2 ...`
+// TRANSLATION MISSING
+===== Functions
+
+Functions are called with `weechat::xxx arg1 arg2 ...`.
 
 ==== Guile (Scheme)
 
-* Le funzioni sono chiamate con `(weechat:xxx arg1 arg2 ...)`
+// TRANSLATION MISSING
-* Le seguenti funzioni prendono un elenco di argomenti (invece di più argomenti
+===== Functions
-  come per le altre funzioni), poiché il numero di argomenti eccede il numero
+
-  di argomenti consentiti in Guile:
+Functions are called with `(weechat:xxx arg1 arg2 ...)`.
-** config_new_section
+
-** config_new_option
+The following functions take one list of arguments (instead of many arguments
-** bar_new
+for other functions), because number of arguments exceed number of allowed
+arguments in Guile:
+
+* config_new_section
+* config_new_option
+* bar_new
 
 ==== JavaScript
 
-* Le funzioni sono chiamate con `weechat.xxx(arg1, arg2, ...);`
+// TRANSLATION MISSING
+===== Functions
+
+Functions are called with `weechat.xxx(arg1, arg2, ...);`.
 
 ==== PHP
 
-* Le funzioni sono chiamate con `weechat_xxx(arg1, arg2, ...);`
+// TRANSLATION MISSING
+===== Functions
+
+Functions are called with `weechat_xxx(arg1, arg2, ...);`.
 
 [[register_function]]
 === Registrare una funzione
@@ -1114,15 +1204,25 @@ weechat.prnt("", "value of option weechat.color.chat_delimiters is: %s"
 [[irc_catch_messages]]
 ==== Catturare messaggi
 
-Il plugin IRC invia due segnali per un messaggio ricevuto (`xxx` è il nome
+// TRANSLATION MISSING
-interno del server IRC, `yyy` è il nome del comando IRC come JOIN, QUIT,
+IRC plugin sends four signals for a message received (`xxx` is IRC internal
-PRIVMSG, 301, ..):
+server name, `yyy` is IRC command name like JOIN, QUIT, PRIVMSG, 301, ..):
 
-xxxx,irc_in_yyy::
+// TRANSLATION MISSING
-    segnale inviato prima di esaminare il messaggio
+xxx,irc_in_yyy::
+    signal sent before processing message, only if message is *not* ignored
 
+// TRANSLATION MISSING
 xxx,irc_in2_yyy::
-    segnale inviato dopo aver esaminato il messaggio
+    signal sent after processing message, only if message is *not* ignored
+
+// TRANSLATION MISSING
+xxx,irc_raw_in_yyy::
+    signal sent before processing message, even if message is ignored
+
+// TRANSLATION MISSING
+xxx,irc_raw_in2_yyy::
+    signal sent after processing message, even if message is ignored
 
 [source,python]
 ----
@@ -1144,8 +1244,19 @@ weechat.hook_signal("*,irc_in2_join", "join_cb", "")
 [[irc_modify_messages]]
 ==== Modificare i messaggi
 
-Il plugin IRC invia un "modificatore" chiamato "irc_in_xxx" ("xxx" è il comando
+// TRANSLATION MISSING
-IRC) per un messaggio ricevuto, in modo da poterlo modificare.
+IRC plugin sends two "modifiers" for a message received ("xxx" is IRC command),
+so that you can modify it:
+
+// TRANSLATION MISSING
+irc_in_xxx::
+    modifier sent before charset decoding: use with caution, the string may
+    contain invalid UTF-8 data; use only for raw operations on a message
+
+// TRANSLATION MISSING
+irc_in2_xxx::
+    modifier sent after charset decoding, so the string received is always
+    UTF-8 valid (*recommended*)
 
 [source,python]
 ----
@@ -1154,7 +1265,7 @@ def modifier_cb(data, modifier, modifier_data, string):
     # (ok non è molto utile, ma è solo un esempio!)
     return "%s %s" % (string, modifier_data)
 
-weechat.hook_modifier("irc_in_privmsg", "modifier_cb", "")
+weechat.hook_modifier("irc_in2_privmsg", "modifier_cb", "")
 ----
 
 [WARNING]

doc/ja/weechat_plugin_api.ja.adoc

@@ -9844,15 +9844,21 @@ WeeChat とプラグインが送信するシグナルのリスト:
   サーバに送信する IRC メッセージ
   (自動分割前、自動分割はデフォルトでメッセージを 512 バイト内に収まるように分割します)。
 
+// TRANSLATION MISSING
 | irc | xxx,irc_out_yyy ^(1)^ |
   String: メッセージ |
   サーバに送信する IRC メッセージ
-  (自動分割後、自動分割はデフォルトでメッセージを 512 バイト内に収まるように分割します)。
+  (自動分割後、自動分割はデフォルトでメッセージを 512 バイト内に収まるように分割します)。 +
+  *Warning:* the string may contain invalid UTF-8 data.
+  Signal "xxx,irc_out1_yyy" is recommended instead.
 
+// TRANSLATION MISSING
 | irc | xxx,irc_outtags_yyy ^(1)^ +
   _(WeeChat バージョン 0.3.4 以上で利用可)_ |
   String: タグ  + ";" + メッセージ |
-  タグ + サーバに送信する IRC メッセージ
+  タグ + サーバに送信する IRC メッセージ +
+  *Warning:* the string may contain invalid UTF-8 data.
+  Signal "xxx,irc_out1_yyy" is recommended instead.
 
 | irc | irc_ctcp |
   String: メッセージ |
@@ -11188,9 +11194,13 @@ WeeChat とプラグインが使う修飾子のリスト:
 |===
 | 修飾子 | 修飾子データ | 文字列 | 出力
 
+// TRANSLATION MISSING
 | [[hook_modifier_irc_in_xxx]] irc_in_xxx ^(1)^ |
   サーバ名 |
-  IRC サーバから受信したメッセージの内容 (文字セットをデコードする前) |
+  IRC サーバから受信したメッセージの内容 (文字セットをデコードする前) +
+  *Warning:* the string may contain invalid UTF-8 data; use only for raw
+  operations on a message.
+  Modifier <<hook_modifier_irc_in2_xxx,irc_in2_xxx>> is recommended instead. |
   メッセージの新しい内容
 
 | [[hook_modifier_irc_in2_xxx]] irc_in2_xxx ^(1)^ +

doc/ja/weechat_scripting.ja.adoc

@@ -3,9 +3,10 @@
 :email: flashcode@flashtux.org
 :lang: ja
 :toc: left
-:toclevels: 3
+:toclevels: 4
 :toc-title: 目次
 :sectnums:
+:sectnumlevels: 3
 :docinfo1:
 
 
@@ -73,22 +74,95 @@ link:weechat_plugin_api.ja.html#_hook_process[WeeChat プラグイン API リフ
 
 ==== Python
 
-* 必ず `import weechat` を使ってください。
+// TRANSLATION MISSING
-* python では `+print*+` 系の関数は `+prnt*+` と書きます
+===== Module
-  (_print_ は予約済みキーワードなので)。
+
-* 関数は `weechat.xxx(arg1, arg2, ...)` のように呼び出してください。
+WeeChat defines a `weechat` module which must be imported with `import weechat`.
+
+// TRANSLATION MISSING
+===== Functions
+
+Functions are called with `weechat.xxx(arg1, arg2, ...)`.
+
+Functions `+print*+` are called `+prnt*+` in python (because `print` was a
+reserved keyword in Python 2).
+
+// TRANSLATION MISSING
+===== Strings received in callbacks
+
+In Python 3 and with WeeChat ≥ 2.7, the strings received in callbacks have type
+`str` if the string has valid UTF-8 data (which is the most common case),
+or `bytes` if the string is not UTF-8 valid. So the callback should take care
+about this type if some invalid UTF-8 content can be received.
+
+Some invalid UTF-8 data may be received in these cases, so the callback can
+receive a string of type `str` or `bytes` (this list is not exhaustive):
+
+[width="100%",cols="3m,3m,3m,8",options="header"]
+|===
+| API function | Arguments | Examples | Description
+
+| hook_modifier |
+  irc_in_yyy |
+  pass:[irc_in_privmsg] +
+  pass:[irc_in_notice] |
+  A message received in IRC plugin, before it is decoded to UTF-8 (used
+  internally). +
+  +
+  It is recommended to use modifier `irc_in2_yyy` instead, the string received
+  is always UTF-8 valid. +
+  See function `hook_modifier` in the
+  link:weechat_plugin_api.en.html#_hook_modifier[WeeChat plugin API reference].
+
+| hook_signal |
+  xxx,irc_out_yyy +
+  xxx,irc_outtags_yyy |
+  pass:[*,irc_out_privmsg] +
+  pass:[*,irc_out_notice] +
+  pass:[*,irc_outtags_privmsg] +
+  pass:[*,irc_outtags_notice] |
+  A message sent by IRC plugin, after it is encoded to the `encode` charset
+  defined by the user (if different from the default `UTF-8`). +
+  +
+  It is recommended to use signal `xxx,irc_out1_yyy` instead, the string received
+  is always UTF-8 valid. +
+  See function `hook_signal` in the
+  link:weechat_plugin_api.en.html#_hook_signal[WeeChat plugin API reference].
+
+| hook_process +
+  hook_process_hashtable |
+  - |
+  - |
+  Output of the command, sent to the callback, can contain invalid UTF-8 data.
+
+|===
+
+In Python 2, which is now deprecated and should not be used any more, the
+strings sent to callbacks were always of type `str`, and may contain invalid
+UTF-8 data, in the cases mentioned above.
 
 ==== Perl
 
-* 関数は `weechat::xxx(arg1, arg2, ...);` のように呼び出してください。
+// TRANSLATION MISSING
+===== Functions
+
+Functions are called with `weechat::xxx(arg1, arg2, ...);`.
 
 ==== Ruby
 
-* _weechat_init_ を定義して、内部で _register_ を呼び出してください。
+// TRANSLATION MISSING
-* 関数は `Weechat.xxx(arg1, arg2, ...)` のように呼び出してください。
+===== Initialization
-* Ruby では関数に渡せる引数の数が最大 15 個に制限されているため、`Weechat.config_new_option`
+
-  関数はコールバック用の引数群を 6 個の文字列からなる 1 個の配列で受け取ります (3 個のコールバック
+You have to define _weechat_init_ and call _register_ inside.
-  + 3 個のデータ文字列)、したがって `Weechat.config_new_option` 関数を呼び出すには以下のようにしてください:
+
+// TRANSLATION MISSING
+===== Functions
+
+Functions are called with `Weechat.xxx(arg1, arg2, ...)`.
+
+Due to a limitation of Ruby (15 arguments max by function), the function
+`Weechat.config_new_option` receives the callbacks in an array of 6 strings
+(3 callbacks + 3 data strings), so a call to this function looks like:
 
 [source,ruby]
 ----
@@ -98,29 +172,46 @@ Weechat.config_new_option(config, section, "name", "string", "description of opt
 
 ==== Lua
 
-* 関数は `weechat.xxx(arg1, arg2, ...)` のように呼び出してください。
+// TRANSLATION MISSING
+===== Functions
+
+Functions are called with `weechat.xxx(arg1, arg2, ...)`.
 
 ==== Tcl
 
-* 関数は `weechat::xxx arg1 arg2 ...` のように呼び出してください。
+// TRANSLATION MISSING
+===== Functions
+
+Functions are called with `weechat::xxx arg1 arg2 ...`.
 
 ==== Guile (Scheme)
 
-* 関数は `(weechat:xxx arg1 arg2 ...)` のように呼び出してください。
+// TRANSLATION MISSING
-* 以下の関数は引数のリストをひとつだけ取ります
+===== Functions
-  (他の関数のように多くの引数を取れません)、この理由は引数の個数が
+
-  Guile で利用できる引数の数を超えるからです。
+Functions are called with `(weechat:xxx arg1 arg2 ...)`.
-** config_new_section
+
-** config_new_option
+The following functions take one list of arguments (instead of many arguments
-** bar_new
+for other functions), because number of arguments exceed number of allowed
+arguments in Guile:
+
+* config_new_section
+* config_new_option
+* bar_new
 
 ==== JavaScript
 
-* 関数は `weechat.xxx(arg1, arg2, ...);` のように呼び出してください。
+// TRANSLATION MISSING
+===== Functions
+
+Functions are called with `weechat.xxx(arg1, arg2, ...);`.
 
 ==== PHP
 
-* 関数は `weechat_xxx(arg1, arg2, ...);` のように呼び出してください。
+// TRANSLATION MISSING
+===== Functions
+
+Functions are called with `weechat_xxx(arg1, arg2, ...);`.
 
 [[register_function]]
 === 関数の登録
@@ -1089,14 +1180,25 @@ weechat.prnt("", "value of option weechat.color.chat_delimiters is: %s"
 [[irc_catch_messages]]
 ==== メッセージのキャッチ
 
-メッセージを受信すると IRC プラグインは 2 つのシグナルを送信します (`xxx`
+// TRANSLATION MISSING
-は IRC 内部サーバ名で、`yyy` は JOIN、QUIT、PRIVMSG、301 等の IRC コマンド名です):
+IRC plugin sends four signals for a message received (`xxx` is IRC internal
+server name, `yyy` is IRC command name like JOIN, QUIT, PRIVMSG, 301, ..):
 
-xxxx,irc_in_yyy::
+// TRANSLATION MISSING
-    メッセージの処理が行われる前に送信されるシグナル
+xxx,irc_in_yyy::
+    signal sent before processing message, only if message is *not* ignored
 
+// TRANSLATION MISSING
 xxx,irc_in2_yyy::
-    メッセージの処理が行われた後に送信されるシグナル
+    signal sent after processing message, only if message is *not* ignored
+
+// TRANSLATION MISSING
+xxx,irc_raw_in_yyy::
+    signal sent before processing message, even if message is ignored
+
+// TRANSLATION MISSING
+xxx,irc_raw_in2_yyy::
+    signal sent after processing message, even if message is ignored
 
 [source,python]
 ----
@@ -1118,8 +1220,19 @@ weechat.hook_signal("*,irc_in2_join", "join_cb", "")
 [[irc_modify_messages]]
 ==== メッセージの修正
 
-メッセージを受信すると IRC プラグインは "irc_in_xxx" ("xxx" は IRC コマンド)
+// TRANSLATION MISSING
-と呼ばれる "modifier" を送信します。メッセージを修正するにはこのシグナルを使います。
+IRC plugin sends two "modifiers" for a message received ("xxx" is IRC command),
+so that you can modify it:
+
+// TRANSLATION MISSING
+irc_in_xxx::
+    modifier sent before charset decoding: use with caution, the string may
+    contain invalid UTF-8 data; use only for raw operations on a message
+
+// TRANSLATION MISSING
+irc_in2_xxx::
+    modifier sent after charset decoding, so the string received is always
+    UTF-8 valid (*recommended*)
 
 [source,python]
 ----
@@ -1128,7 +1241,7 @@ def modifier_cb(data, modifier, modifier_data, string):
     # (これは役に立ちませんが、例として!)
     return "%s %s" % (string, modifier_data)
 
-weechat.hook_modifier("irc_in_privmsg", "modifier_cb", "")
+weechat.hook_modifier("irc_in2_privmsg", "modifier_cb", "")
 ----
 
 [WARNING]

doc/pl/weechat_scripting.pl.adoc

@@ -3,9 +3,10 @@
 :email: flashcode@flashtux.org
 :lang: pl
 :toc: left
-:toclevels: 3
+:toclevels: 4
 :toc-title: Spis treści
 :sectnums:
+:sectnumlevels: 3
 :docinfo1:
 
 
@@ -73,54 +74,144 @@ link:weechat_plugin_api.en.html#_hook_process[Opisu API wtyczek WeeChat] (Angiel
 
 ==== Python
 
-* Należy wykonać `import weechat`
+// TRANSLATION MISSING
-* Funkcje `+print*+` są nazwane `+prnt*+` w pythonie (ponieważ _print_ jest zastrzeżonym
+===== Module
-  słowem kluczowym)
+
-* Funkcje są wywoływane za pomocą `weechat.xxx(arg1, arg2, ...)`
+WeeChat defines a `weechat` module which must be imported with `import weechat`.
+
+// TRANSLATION MISSING
+===== Functions
+
+Functions are called with `weechat.xxx(arg1, arg2, ...)`.
+
+Functions `+print*+` are called `+prnt*+` in python (because `print` was a
+reserved keyword in Python 2).
+
+// TRANSLATION MISSING
+===== Strings received in callbacks
+
+In Python 3 and with WeeChat ≥ 2.7, the strings received in callbacks have type
+`str` if the string has valid UTF-8 data (which is the most common case),
+or `bytes` if the string is not UTF-8 valid. So the callback should take care
+about this type if some invalid UTF-8 content can be received.
+
+Some invalid UTF-8 data may be received in these cases, so the callback can
+receive a string of type `str` or `bytes` (this list is not exhaustive):
+
+[width="100%",cols="3m,3m,3m,8",options="header"]
+|===
+| API function | Arguments | Examples | Description
+
+| hook_modifier |
+  irc_in_yyy |
+  pass:[irc_in_privmsg] +
+  pass:[irc_in_notice] |
+  A message received in IRC plugin, before it is decoded to UTF-8 (used
+  internally). +
+  +
+  It is recommended to use modifier `irc_in2_yyy` instead, the string received
+  is always UTF-8 valid. +
+  See function `hook_modifier` in the
+  link:weechat_plugin_api.en.html#_hook_modifier[WeeChat plugin API reference].
+
+| hook_signal |
+  xxx,irc_out_yyy +
+  xxx,irc_outtags_yyy |
+  pass:[*,irc_out_privmsg] +
+  pass:[*,irc_out_notice] +
+  pass:[*,irc_outtags_privmsg] +
+  pass:[*,irc_outtags_notice] |
+  A message sent by IRC plugin, after it is encoded to the `encode` charset
+  defined by the user (if different from the default `UTF-8`). +
+  +
+  It is recommended to use signal `xxx,irc_out1_yyy` instead, the string received
+  is always UTF-8 valid. +
+  See function `hook_signal` in the
+  link:weechat_plugin_api.en.html#_hook_signal[WeeChat plugin API reference].
+
+| hook_process +
+  hook_process_hashtable |
+  - |
+  - |
+  Output of the command, sent to the callback, can contain invalid UTF-8 data.
+
+|===
+
+In Python 2, which is now deprecated and should not be used any more, the
+strings sent to callbacks were always of type `str`, and may contain invalid
+UTF-8 data, in the cases mentioned above.
 
 ==== Perl
 
-* Funkcje są wywoływane za pomocą `weechat::xxx(arg1, arg2, ...);`
+// TRANSLATION MISSING
+===== Functions
+
+Functions are called with `weechat::xxx(arg1, arg2, ...);`.
 
 ==== Ruby
 
-* Trzeba zdefiniować _weechat_init_ i wywołać _register_ wewnątrz
+// TRANSLATION MISSING
-* Funkcje są wywoływane za pomocą `Weechat.xxx(arg1, arg2, ...)`
+===== Initialization
-* W związku z ograniczeniami Ruby (maksymalnie 15 argumentów dla funkcji), funkcja
+
-  `Weechat.config_new_option` otrzymuje callbacki w postaci tablicy 6 ciągów
+You have to define _weechat_init_ and call _register_ inside.
-  (3 callbacki + 3 ciągi danych), wywołanie tej funkcji wygląda następująco:
+
+// TRANSLATION MISSING
+===== Functions
+
+Functions are called with `Weechat.xxx(arg1, arg2, ...)`.
+
+Due to a limitation of Ruby (15 arguments max by function), the function
+`Weechat.config_new_option` receives the callbacks in an array of 6 strings
+(3 callbacks + 3 data strings), so a call to this function looks like:
 
 [source,ruby]
 ----
-Weechat.config_new_option(config, section, "nazwa", "ciąg", "opis opcji", "", 0, 0,
+Weechat.config_new_option(config, section, "name", "string", "description of option", "", 0, 0,
-                          "wartość", "wartość", 0, ["check_cb", "", "change_cb", "", "delete_cb", ""])
+                          "value", "value", 0, ["check_cb", "", "change_cb", "", "delete_cb", ""])
 ----
 
 ==== Lua
 
-* Funkcje są wywoływane za pomocą `weechat.xxx(arg1, arg2, ...)`
+// TRANSLATION MISSING
+===== Functions
+
+Functions are called with `weechat.xxx(arg1, arg2, ...)`.
 
 ==== Tcl
 
-* Funkcje są wywoływane za pomocą `weechat::xxx arg1 arg2 ...`
+// TRANSLATION MISSING
+===== Functions
+
+Functions are called with `weechat::xxx arg1 arg2 ...`.
 
 ==== Guile (Scheme)
 
-* Funkcje są wywoływane za pomocą `(weechat:xxx arg1 arg2 ...)`
+// TRANSLATION MISSING
-* Następujące funkcje przyjmują pojedynczą listę argumentów (zamiast wielu
+===== Functions
-  argumentów dla innych funkcji), ponieważ ilość argumentów przekracza ilość
+
-  argumentów dozwolonych w Guile:
+Functions are called with `(weechat:xxx arg1 arg2 ...)`.
-** config_new_section
+
-** config_new_option
+The following functions take one list of arguments (instead of many arguments
-** bar_new
+for other functions), because number of arguments exceed number of allowed
+arguments in Guile:
+
+* config_new_section
+* config_new_option
+* bar_new
 
 ==== JavaScript
 
-* Funkcje są wywoływane za pomocą `weechat.xxx(arg1, arg2, ...);`
+// TRANSLATION MISSING
+===== Functions
+
+Functions are called with `weechat.xxx(arg1, arg2, ...);`.
 
 ==== PHP
 
-* Funkcje są wywoływane za pomocą `weechat_xxx(arg1, arg2, ...);`
+// TRANSLATION MISSING
+===== Functions
+
+Functions are called with `weechat_xxx(arg1, arg2, ...);`.
 
 [[register_function]]
 === Funkcja rejestrująca
@@ -1087,14 +1178,25 @@ weechat.prnt("", "wartość opcji weechat.color.chat_delimiters to: %s"
 [[irc_catch_messages]]
 ==== Przechwytywanie wiadomości
 
-Wtyczka IRC wysyła dwa sygnały dla otrzymanej wiadomości (`xxx` jest wewnętrzną
+// TRANSLATION MISSING
- nazwą serwera IRC, `yyy` to komenda IRC jak JOIN, QUIT, PRIVMSG, 301, ..):
+IRC plugin sends four signals for a message received (`xxx` is IRC internal
+server name, `yyy` is IRC command name like JOIN, QUIT, PRIVMSG, 301, ..):
 
-xxxx,irc_in_yyy::
+// TRANSLATION MISSING
-    sygnał wysłany przed przetworzeniem wiadomości
+xxx,irc_in_yyy::
+    signal sent before processing message, only if message is *not* ignored
 
+// TRANSLATION MISSING
 xxx,irc_in2_yyy::
-    sygnał wysłany po przetworzeniu wiadomości
+    signal sent after processing message, only if message is *not* ignored
+
+// TRANSLATION MISSING
+xxx,irc_raw_in_yyy::
+    signal sent before processing message, even if message is ignored
+
+// TRANSLATION MISSING
+xxx,irc_raw_in2_yyy::
+    signal sent after processing message, even if message is ignored
 
 [source,python]
 ----
@@ -1116,8 +1218,19 @@ weechat.hook_signal("*,irc_in2_join", "join_cb", "")
 [[irc_modify_messages]]
 ==== Modyfikowanie wiadomości
 
-Wtyczka IRC wysyła "modyfikator" nazwany "irc_in_xxx" ("xxx" to komenda IRC) dla
+// TRANSLATION MISSING
-otrzymanej wiadomości, żeby można było ją zmodyfikować.
+IRC plugin sends two "modifiers" for a message received ("xxx" is IRC command),
+so that you can modify it:
+
+// TRANSLATION MISSING
+irc_in_xxx::
+    modifier sent before charset decoding: use with caution, the string may
+    contain invalid UTF-8 data; use only for raw operations on a message
+
+// TRANSLATION MISSING
+irc_in2_xxx::
+    modifier sent after charset decoding, so the string received is always
+    UTF-8 valid (*recommended*)
 
 [source,python]
 ----
@@ -1126,7 +1239,7 @@ def modifier_cb(data, modifier, modifier_data, string):
     # (nie jest to może bardzo przydatne, ale to tylko przykład!)
     return "%s %s" % (string, modifier_data)
 
-weechat.hook_modifier("irc_in_privmsg", "modifier_cb", "")
+weechat.hook_modifier("irc_in2_privmsg", "modifier_cb", "")
 ----
 
 [WARNING]

src/plugins/python/weechat-python.c

@@ -446,12 +446,13 @@ weechat_python_output (PyObject *self, PyObject *args)
 void *
 weechat_python_exec (struct t_plugin_script *script,
                      int ret_type, const char *function,
-                     char *format, void **argv)
+                     const char *format, void **argv)
 {
     struct t_plugin_script *old_python_current_script;
     PyThreadState *old_interpreter;
     PyObject *evMain, *evDict, *evFunc, *rc;
     void *argv2[16], *ret_value, *ret_temp;
+    char format2[17];
     int i, argc, *ret_int;
 
     ret_value = NULL;
@@ -485,9 +486,33 @@ weechat_python_exec (struct t_plugin_script *script,
         argc = strlen (format);
         for (i = 0; i < 16; i++)
         {
-            argv2[i] = (i < argc) ? argv[i] : NULL;
+            if (i < argc)
+            {
+                argv2[i] = argv[i];
+                if (format[i] == 's')
+                {
+#if PY_MAJOR_VERSION >= 3
+                    if (weechat_utf8_is_valid (argv2[i], -1, NULL))
+                        format2[i] = 's';  /* Python 3: str */
+                    else
+                        format2[i] = 'y';  /* Python 3: bytes */
+#else
+                    format2[i] = 's';  /* Python 2: str */
+#endif
+                }
+                else
+                {
+                    format2[i] = format[i];
+                }
+            }
+            else
+            {
+                argv2[i] = NULL;
+            }
         }
-        rc = PyObject_CallFunction (evFunc, format,
+        format2[argc] = '\0';
+
+        rc = PyObject_CallFunction (evFunc, format2,
                                     argv2[0], argv2[1],
                                     argv2[2], argv2[3],
                                     argv2[4], argv2[5],

src/plugins/python/weechat-python.h

@@ -61,6 +61,6 @@ extern struct t_hashtable *weechat_python_dict_to_hashtable (PyObject *dict,
                                                              const char *type_values);
 extern void *weechat_python_exec (struct t_plugin_script *script,
                                   int ret_type, const char *function,
-                                  char *format, void **argv);
+                                  const char *format, void **argv);
 
 #endif /* WEECHAT_PLUGIN_PYTHON_H */