Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

miyagawa (1653)

miyagawa
  (email not shown publicly)
http://bulknews.vox.com/
AOL IM: bulknews (Add Buddy, Send Message)

Journal of miyagawa (1653)

Tuesday April 07, 2009
06:51 PM

DBD::SQLite and Unicode

Attention to anyone using DBD::SQLite and $dbh->{unicode} attribute set to 1.

This module has a long standing bug where it assumes passed strings internal encoding is UTF-8 when inserting values into the database and I'm trying to fix it.


use DBI;
use Encode;

my $utf8_string = "This is \x{30c6}\x{30b9}\x{30c8}"; # "Test" in Japanese
my $utf8_bytes = encode_utf8($string);
my $lat1_string = "H\xe9llo World"; # Héllo

my $dbh = DBI->connect("DBI:SQLite:...", ...);
$dbh->{unicode} = 1;

my $sth = $dbh->prepare("INSERT INTO foo (bar) VALUES (?)";

$sth->execute($utf8_string); # (1) Good
$sth->execute($utf8_bytes); # (2) ???
$sth->execute($lat1_string); # (3) ???

Current version of DBD::SQLite (prior to 1.21) assumes given string's INTERNAL encoding as UTF-8 and stores the octet stream into the database without calling encode_utf8 nor utf8::upgrade, so this makes #2 PASS and #3 FAIL (invalid UTF-8 octet in the database), which is not correct.

My patch solves this, and #2 $utf8_bytes will be now double encoded and FAIL, but #3 PASS with correct UTF-8 octet stream.

That #2 FAIL might break your (potentially-already-broken) app, when you try to save UTF-8 encoded strings into the database under 'unicode' option, but I believe this is a right fix to make it FAIL.

http://svn.ali.as/cpan/trunk/DBD-SQLite/t/rt_25371_asymmetric_unicode.t is a failing test by Juerd and http://fisheye2.atlassian.com/changelog/cpan/trunk/DBD-SQLite?cs=6077 is my patch to fix that. This patch still passes all tests, including 12_unicode.t and 20_blobs.t, and this makes DBD::SQLite's unicode option compatible to what DBD::mysql's mysql_enable_utf8 option does, etc.

Note that if you REALLY want to save the octet bytes without being encoded into UTF-8, you can still define the table with BLOB column type and use 3-arg bind_param like explained in DBD::SQLite POD. That 'unicode' section continues to be entirely correct with this patch.

Let me know your input in #dbd-sqlite on irc.perl.org. Testing your app with my patch and reporting it back would be highly appreciated too.

Thursday January 15, 2009
12:57 PM

CPAN Timeline

Have a Google account and Perl hacker friends? Try CPAN Timeline and see what your friends are hacking on. This could be a proof-of-concept app how to make CPAN more social place. The other approach using the existent social network includes Leon's facebook CPAN app.
Tuesday November 25, 2008
02:45 PM

Shibuya.pm Tech Meeting #10

Shibuya Perl Mongers tech meeting #10 is full of interesting talks and is scheduled on November 27th, Thursday 18:30 in Japan time. There will be ustream.tv streaming available for all talks.

I will give a short talk and demo about my new toy remedie. This is my answer to the recent video sharing and tv shows buzz over the internet, using my other powerful tool Plagger. I already can't live without this app and I hope I'll talk about it in YAPCs (Asia, NA, EU) and OSCON.

Thursday July 03, 2008
03:01 PM

Hawaii, where YAPC::Asia, YAPC::NA and OSDC.au can meet

Sorry, I just couldn't resist :)
Wednesday June 18, 2008
09:21 PM

Shibuya.pm: tech talks XS night

Shibuya.pm will have its 9th technical meeting and the topic of the meeting is XS. No, I'm not joking and all the talks are somehow about XSUB stuff. Let me quote some talks:

1. My First XS (hirose31)
2. Welcome to Perl5 Internals (Daisuke Maki)
3. Inside Ruby.pm (Goro Fuji)
4. PerlMachine (wakapon)

PerlMachine is a crazy project that is a minimal linux kernel that is designed solely to run perl. He reimplement most perl built-in functions in XSUBs and link them directly to the kernel functions he made. So I assume it's like lispmachine for Perl.

It is quite exciting that PerlMachine and Ruby.pm talks are done by students from University of Tsukuba. Perl has a bright future! :)

I can't make it to the meeting because I'm back here in San Francisco (so this is the first Shibuya.pm meeting that I can't attend) but I'm sure there will be ustream. Looking forward to it!

Sunday May 18, 2008
08:26 AM

Big thanks to Pauley's as well

Everyone at YAPC::Asia recognizes Dan Kogai as a big contributor to the organizers because he offers his great condo (a.k.a Hotel DAN) to hackers from overseas and also makes the place open for hackathoners. This year Larry and Gloria Wall, jrockway, Yuval, clkao, ingydotnet and Jesse stayed at his place.

I should now mention that I equally thank Marty and Karen Pauley for hosting Jose (cog), Casey West and Michael Schwern. Karen also demonstrated her awesome typing skills by transcribing Larry and Schwern's keynote. I believe that was really helpful to most Japanese audience to grok what they are talking about.

Thank you.

Friday May 16, 2008
10:10 AM

My 20 modules talk at YAPC::Asia

I just started writing my slides midnight before the conference (which is very usual) and trimmed the number of modules I talk about down to 10. I think the talk went really well, especially my evil script to authenticate the wireless access using WWW::Mechanize :)

http://www.slideshare.net/miyagawa/20-modules-i-havent-yet-talked-about/

Wednesday May 14, 2008
11:41 PM

YAPC::Asia 2008 live stream

YAPC::Asia 2008 just started and is in its full swing.

For those who cannot make it, we have a stream available on http://live.yapcasia.org/ thanks to our streaming sponsors.

Enjoy!

Tuesday May 06, 2008
10:02 AM

Looking for YAPC::Asia 2009 organizers

YAPC::Asia has become huge. This year we've got 550 registrations and I think this is one of the biggest YAPCs ever.

However, our organization team has been getting smaller year by year, maybe because we knew we can do this. I live in San Francisco, USA and remotely organize the conference for this 2 years, just like any other project managers do for a project. That means, the real tough work has been done by our staff in Tokyo (I think it's too early to name and thank them since the conference has not even started)

That said, I'd like to hand off this conference to someone else, since I don't have a plan to get back to Japan anytime soon.

We're looking for the organizers for YAPC::Asia 2009. It could be any other individuals or preferrably Perl Mongers group in Asia. We've been writing things down what's needed to run this conference and already started writing Post-Mortem document so that this conference could be even better next year. I could still help configuring Act site, doing public relations to guests (if needed) etc. that I can still do remotely.

Drop me an email (miyagawa at gmail.com) if you're interested. Hopefully we can announce in the closing ceremony in this YAPC::Asia (May 15-16), and people will stop asking me about "When and where is YAPC::Asia 2009?" :)

Thursday May 01, 2008
02:08 PM

YAPC::Asia 2008 schedule is out

The schedule of YAPC::Asia 2008 is now out. We have full of interesting talks for 3 tracks on 2 full days. Pretty exciting.

We also started the call for Lightning Talks today. Propose yours to speak for 5 minutes whatever you want.