javarpc and accented characters

[daredevil2000]

I used a servlet that creates a simple XML file from SQL query.
I used an accented font (arial).
Everything works fine.

I used the same SQL query in a Java method that returns an ArrayList of HashMap.
thru this method I could not get all the accented caracters.

Why ?

[hqm]

There are two bugs in the Java RPC encoder:

1) If you're using LPS 3.0xx, the Java RPC encoder is still writing Flash 5 files, which screws up multibyte UTF8 chars.

2) There's a bug in the data compilation code that will mess up the second time it emits a cached string (this again will fail only in UTF8/Flash 6).

Hopefully fixes for these will make it into the 3.0b2 release, which should be out quite soon. Please run your code on that release and tell me if you still see the bug.

(hminsky@laszlosystems.com)

[pablo]

I just checked in a fix for JavaRPC accented characters which should make it in 3.0b2. You'll also find an example.

pablo

[daredevil2000]

I await impatiently this new version.

[daredevil2000]

Now (with 3.0b2), when the dataset fill the grid, the caracters are displayed correctly.

But, when I modify a line and I write the line modified throught a Java RPC object, I get messy things in the database.

================================================== ===
var vSel = this.getSelection();
this.aColl_id=Number(vSel.xPathQuery("coll_id/text()"));
this.aColl_nom=this.adatap.xPathQuery("coll_nom/text()");
this.aColl_prenom=this.adatap.xPathQuery("coll_prenom/text()");
var vAnnexp=this.adatap.xPathQuery("coll_annexp/text()");
Debug.Write(" gridOnSelect -- annexp :",vAnnexp)
if (vAnnexp=="")
this.aColl_annexp=Number(null);
else
this.aColl_annexp=Number(vAnnexp);
this.aColl_email=this.adatap.xPathQuery("coll_email/text()");
this.aColl_parain=this.adatap.xPathQuery("coll_parain/text()");

CollService.proxy.setCollaborateur([aColl_id,aColl_nom,aColl_prenom,
aColl_annexp,aColl_email,
aColl_parain],CollService.mySetCol);
================================================== ===

[hqm]

Can you send a small test case, including what you typed into the input fields?

I suspect there is a bug in the Java server side decoding of string data, it depends to some extent on which web server you are running.

Are you using Tomcat 5 for your server? On which operating system?

[daredevil2000]

Im a using the version of tomcat provided by the install of laszlo.

The zip contains :
* an Mysql file that creates the tables and fill it.
* lzxCol.zip contains the Lzx files (used in Eclipse).
- lstCol.lzx contains the window
- dataLstCol.lzx contains the scripts and initialize the ata.
* jarColl.zip contains the java files used by dataLstCol.lzx (I generate a jarCol.jar file I put in the common/lib repository of Tomcat).

I select a cell (Nom). I enter 'é' or 'à'.
I then click on another row and I answer 'Yes/Oui' to save the row.

I then restart the lzxCol file to reload the grid.

[pablo]

Looking at this issue. I wrote a simple test case that sends a string containing accented characters up to the server and back. Looks like the string isn't getting encoded correctly upstream. Will send out details as I know more.

Thanks,
pablo

[hqm]

Pablo,

I've found some confusing cases of what seems to be inconsistent behavior of the java JSP servlet with respect to how it decodes UTF-8 data that is in a POST or GET request, there may be something related. Please keep me informed.

Thanks,
Henry

[daredevil2000]

I setup log4J at debug level.
I found that this XML structure is sent to the server :
===============================
21 mars 2005 22:26:27 (127.0.0.1 6) DEBUG data.DataSource - 'url' is java://jarColl.CollService
21 mars 2005 22:26:27 (127.0.0.1 6) DEBUG utils.LZHttpUtils - modifyWEBAPP
21 mars 2005 22:26:27 (127.0.0.1 6) INFO data.DataSource - requesting URL: 'java://jarColl.CollService'
21 mars 2005 22:26:27 (127.0.0.1 6) DEBUG data.JavaDataSource - getData
21 mars 2005 22:26:27 (127.0.0.1 6) DEBUG data.DataSource - 'url' is java://jarColl.CollService
21 mars 2005 22:26:27 (127.0.0.1 6) DEBUG utils.LZHttpUtils - modifyWEBAPP
21 mars 2005 22:26:27 (127.0.0.1 6) DEBUG server.Option - checking: jarColl.CollService
21 mars 2005 22:26:27 (127.0.0.1 6) DEBUG data.JavaDataSource - class name: jarColl.CollService
21 mars 2005 22:26:27 (127.0.0.1 6) DEBUG data.JavaDataSource - object return type: pojo
21 mars 2005 22:26:27 (127.0.0.1 6) DEBUG data.JavaDataSource - POST body:
<?xml version="1.0"?>
<methodCall>
<methodName>jarColl.CollService.setCollaborateur</methodName>
<params>
<param>
<value><i4>472</i4></value>
</param>
<param>
<value><string>EÃ_</string></value>
</param>
<param>
<value><string>e</string></value>
</param>
<param>
<value><i4>0</i4></value>
</param>
<param>
<value><string>t</string></value>
</param>
<param>
<value><null>null</null></value>
</param>
</params>
</methodCall>
21 mars 2005 22:26:27 (127.0.0.1 6) DEBUG data.JavaDataSource - execute(jarColl.CollService.setCollaborateur, [ 472 EÃ_ e 0 t null ],context)
21 mars 2005 22:26:27 (127.0.0.1 6) DEBUG data.JavaDataSource - Searching for method: setCollaborateur in class jarColl.CollService
21 mars 2005 22:26:27 (127.0.0.1 6) DEBUG data.JavaDataSource - Parameter 0: 472 (int)
21 mars 2005 22:26:27 (127.0.0.1 6) DEBUG data.JavaDataSource - Parameter 1: EÃ_ (class java.lang.String)
21 mars 2005 22:26:27 (127.0.0.1 6) DEBUG data.JavaDataSource - Parameter 2: e (class java.lang.String)
21 mars 2005 22:26:27 (127.0.0.1 6) DEBUG data.JavaDataSource - Parameter 3: 0 (int)
21 mars 2005 22:26:27 (127.0.0.1 6) DEBUG data.JavaDataSource - Parameter 4: t (class java.lang.String)
21 mars 2005 22:26:27 (127.0.0.1 6) DEBUG data.JavaDataSource - Parameter 5: null (class java.lang.String)
21 mars 2005 22:26:27 (127.0.0.1 6) DEBUG remote.LZReturnObject - createObject(1)
21 mars 2005 22:26:27 (127.0.0.1 6) DEBUG remote.LZReturnObject - createObjectFile(1, 6)
21 mars 2005 22:26:27 (127.0.0.1 6) DEBUG remote.LZReturnObject - createObjectProgram(1)
21 mars 2005 22:26:27 (127.0.0.1 6) DEBUG remote.LZReturnObject - createReturnValue
21 mars 2005 22:26:27 (127.0.0.1 6) DEBUG remote.LZReturnObject - pushInteger

===========================
The string parameter EÃ is false.
When this XML structure is produced ?
It seems that is the POST body produced by the client.

[daredevil2000]

I found in \lps\components\rpc\library\javarpc.js :

=============================================
var methodname = opts['methodname'];
var mesg = new _root.XMLRPCMessage(methodname);
for (var i=0; i < args.length; i++) {
mesg.addParameter(_root.LzBrowser.xmlEscape(args[i]));
}
==============================================

How does the xmlEscape function work ?

[daredevil2000]

Spying the javarpc.js file I checked that the xml structure sent to the server is OK.

So that's really Tomcat that transform the post message.

How do you specify the enctype of the post ?
example in HTML :
<form action= "/DoSearch " method= "post" target= "content " enctype= "text/plain;charset=UTF-8" >

[pablo]

Not positive.

I spent a whole day trying to figure where the servlet is doing transforming the encoding. I have a test case where I'm sending UTF-8 characters from javarpc method to client. Then I send that same string from the client, where it looks good, back up to server. I log the string when the server receives it and it seems to be doing the wrong thing.

I've been sidetracked by other projects, but I am still looking at this issue. Will keep you informed as I find out more.

pablo

[daredevil2000]

In the log generated by laszlo on the server, the content type of the post is set to "application/x-www-form-urlencoded"


I think it should be set to "application/x-www-form-urlencoded; charset=ISO-8855-P1".

[hqm]

Yes, that would be better. Unfortunately I don't know if there is any API to tell the Flash player to set charset encoding in the content type of the header.

[ehsavoie]

Could'nt we specify the encoding in the Openlaszlo servlet that makes the call ??
A simple new String(parameter.getBytes(encoding), "UTF-8"); is sufficient.

[hqm]

From my quick look at the source code, the problem seemed to be in

org.openlaszlo.remote.LzXMLRPCRequestProcessor which extends
org.apache.xmlrpc.XMLRPCRequestProcessor. XMLRPCRequestProcessor

That is supposed to grab the parameters from the incoming request, and as far as I can tell it is screwing up because the strings are coming up from the Flash player encoded as UTF-8 inside of ISO-8859-1.

The workaround is to grab the raw bytes from the string and re-encode them as UTF-8


http://ws.apache.org/xmlrpc/apidocs/org/apache/xmlrpc/XmlRpcRequestProcessor.html#decodeRequest(java.io. InputStream)

I tried looking at these docs and then compiling something like this



public class LZXmlRpcRequestProcessor extends XmlRpcRequestProcessor
{
public LZXmlRpcRequestProcessor() {
super();
}

public XmlRpcServerRequest decodeRequest(java.io.InputStream is) {
this.setInputEncoding("ISO-8859-1");
return super.decodeRequest(is);
}


}


but the versions of the apache commons rpc library we are using are older and don't support those methods yet. On the roadmap is plans to update to newer versions of the apache libraries.


A workaround is in the methods you are calling to re-encode the strings manually, something like this


public String echo2(String val, Vector vec) {
String ustr = ( String )vec.elementAt(0);
byte p[];
try {
//p = v[i].getBytes("UTF-8"); // WORKS for PROXIED POST
p = ustr.getBytes("ISO-8859-1"); // OK for GET, proxied and direct
String rstr = new String(p, 0, p.length, "UTF-8");

return "val="+val + " :: "+val.length()+" vec[0]="+rstr+"
:: "+ rstr.length();
} catch (Exception e) {
return "caught exception "+e;
}
}

[ehsavoie]

Thanks, I'm not familiar enougth with Laszlo and i thought that the parameters were passed using a simple POST and not a XMLRequest.
I reencoded all my method params before executing my code and it works fine and displays well.