Arbit - project tracking

Arbit - project tracking

#91: German Umlauts throw iconv notices, issues are not saved

Issue attachments

Issue revisions

  • new by Markus Wolff at 2009-O-12 14:21
  • new by Markus Wolff at 2009-O-12 14:21
  • assigned by Kore Nordmann at 2009-O-12 14:57
  • assigned by Kore Nordmann at 2009-N-09 9:21
  • assigned by Kore Nordmann at 2009-N-09 9:21
  • closed by Kore Nordmann at 2009-N-12 7:57
Type bug bug
State closed closed
Priority urgent urgent
Resolution fixed fixed
Assigned to Kore Nordmann
Scheduled for 0.3-alpha
Affected versions Latest SVN
Affected components Modules / Tracker
Last change Thursday 12 November 2009 07:57:15 UTC by Kore Nordmann

Short description

When trying to use German Umlaut characters within an issue title or description, a notice about illegal characters is presented.

A second message claims that the issue has been saved correctly, but it does not show up in the issue list or the overview screen.

Environment

Ubuntu 9.04 Server 32bit, PHP 5.3.0, CouchDB 0.9

Steps to reproduce

  • Create a new issue

  • Enter "täst" or something similar into the title field

  • Save the issue

(please substitute the entity ä with the proper Umlaut character)

Expected behavior

No notices should be thrown, the issue should be saved

Actual behavior

A notice is displayed:

A PHP error occured: Notice: iconv() [<a href='function.iconv'>function.iconv</a>]: Detected an illegal character in input string

The issue is not being saved.

  • Kore Nordmann at Monday 12 October 2009 14:38:17 UTC

    A first investigation shows that the problem occurs in the Zend_Search_Lucene text analyser. Switching the search backend to Solr should therefor help.

    I will see how we can workaround this, anyways.

    Problem located at: libraries/Zend/Search/Lucene/Analysis/Analyzer/Common/Text.php line 58.

  • Kore Nordmann at Monday 12 October 2009 14:56:36 UTC

    Since ZSL uses an empty encoding by default, and does not use the set input encoding from iconv as a default value, the encoding handling does not work there.

    The attached patch sets the encoding explicitly in the ezcSearch handler. The patch will be reported upstream. For now the patch is also available as libraries/ezc_search_encoding.patch since revision #1369 in trunk.

  • Markus Wolff at Monday 12 October 2009 21:05:30 UTC

    Patch works like a charm - thanks a lot, Kore :-)

  • Markus at Monday 2 November 2009 15:54:04 UTC

    Stumbled upon a related issue, not sure if it warrants its own ticket as the solution is probably very similar: When I use the fulltext search function and enter a special character, e.g. "รค", an exception is thrown.

    exception 'arbitPhpErrorException' with message 'A PHP error occured: Notice: iconv_strlen() [<a href='function.iconv-strlen'>function.iconv-strlen</a>]: Detected an illegal character in input string' in /src/libraries/Zend/Search/Lucene/Search/QueryLexer.php:342

    Stack trace: #0 [internal function]: arbitErrorHandler(8, 'iconv_strlen() ...', '/src...', 342, Array) #1 /src/libraries/Zend/Search/Lucene/Search/QueryLexer.php(342): iconv_strlen('ezcsearch_type:...', '') #2 /src/libraries/Zend/Search/Lucene/Search/QueryParser.php(387): Zend_Search_Lucene_Search_QueryLexer->tokenize('ezcsearch_type:...', '') #3 /src/libraries/Zend/Search/Lucene.php(891): Zend_Search_Lucene_Search_QueryParser::parse('ezcsearch_type:...') #4 [internal function]: Zend_Search_Lucene->find('ezcsearch_type:...') #5 /src/libraries/Zend/Search/Lucene/Proxy.php(346): call_user_func_array(Array, Array) #6 [internal function]: Zend_Search_Lucene_Proxy->find('ezcsearch_type:...') #7 /src/libraries/ezc/search/handlers/zend_lucene.php(196): call_user_func_array(Array, Array) #8 /src/libraries/ezc/search/handlers/zend_lucene.php(251): ezcSearchZendLuceneHandler->search('ezcsearch_type:...', Array, NULL) #9 /src/libraries/ezc/search/search_session.php(70): ezcSearchZendLuceneHandler->find(Object(ezcSearchQueryZendLucene)) #10 /src/modules/tracker/classes/controller.php(290): ezcSearchSession->find(Object(ezcSearchQueryZendLucene)) #11 /src/classes/controller/project.php(70): arbitModuleTrackerController->search(Object(arbitHttpRequest)) #12 /src/classes/controller/project.php(188): arbitProjectController->callModuleController('bugs', Object(arbitHttpRequest)) #13 /src/libraries/ezc/mvc/dispatchers/configurable.php(246): arbitProjectController->createResult() #14 /src/htdocs/index.php(76): ezcMvcConfigurableDispatcher->run() #15 {main}

  • Kore Nordmann at Monday 9 November 2009 09:21:53 UTC

    Scheduled for 0.3-alpha.

  • Kore Nordmann at Thursday 12 November 2009 07:55:46 UTC

    Updated patch to cover both issues.

  • Kore Nordmann at Thursday 12 November 2009 07:57:15 UTC

    Fixed by available patch in #1504.

    Upstream release including patch pending.

  • Kore Nordmann at Thursday 12 November 2009 08:56:43 UTC

    Bugs reported upstream:

Arbit - project tracking, licensed under GPL 3