I have enough entries now at Random Observations that I'm fairly sure I'll post at least sporadically. Be warned that little of what I've felt motivated to write about so far is Perl related. Though I'm sure I'll write some Perl stuff eventually.
For personal reasons I've been writing up an explanation of the Kelly Criterion. That is a rule for how to optimize your betting strategy if you find a bet that is favorable to you. However I want to go beyond the rule (which is well-known by gamblers) and get at more subtle issues caused by complicated betting outcomes, choices of multiple bets, and worrying about how to trade off short term volatility with long term returns.
The explanation isn't done yet and the calculator doesn't have the optimization feature that I really want to add. However I have enough there to look for feedback.
Please keep in mind that I am targeting gamblers who are comfortable playing with numbers and know something about probability, but whose advanced math skills may be rusty to non-existent.
I have long repeated the maxim that you write in a high level language, and only after it proves too slow do you consider dropping to a low level language. I have also found that the occasions where it is appropriate to go low level are few and far between.
In my past experience, I never felt the need. Generally most of the stuff that I work on is I/O bound. And the rest is stuff which it is cheaper to throw another server at than to take programmer time.
But that changed. While working on a contract I encountered the need. I had specced out an algorithm to do a particular task, optimized it somewhat, and implemented it as a series of MySQL queries. With some tweaking I got it to run the job I needed with the plan I wanted, but it was still going to take 5 days. I could extract some naive parallelism, but that would only take it down to a day or so. Since they want the job to run regularly, I need to do better than that. So I decided to go with C++.
The computation dropped from 5 days to 10 minutes! Apparently all of the following are good for speed: Manipulating smaller data structures, being able to fit key data structures into level 1 and 2 cache rather than struggling to keep it in RAM, using direct compiled code rather than microcode, avoiding sorts by keeping things in the order I need, and having direct array lookups rather than searching btree indexes. There were some other minor tweaks done, and more I could do, but on the whole I am quite satisfied with the result.
(And, for C++, it is fairly readable.)
I just encountered this problem on MySQL. But don't blame the database, since I've seen similar misbehavior on PostgreSQL and Oracle.
Someone put together a reporting query. In it there are many left joins to subqueries. The overall query was painfully slow and getting slower over time. I was asked to improve this.
The solution is to move all of the subqueries into queries that populate temp tables. Put indexes on those temp tables. Then do the big join and watch it run much faster.
The reason why this works is that this plan is not in the query optimizer's repertoire. And the reason for that is that if the query optimizer tried to analyze every possible strategy for a complex query, it could take longer to run the analyzer than to run the query!
Still this pattern does come up from time to time, so it is a good trick to have in your toolbox.
I haven't been sharing my life, but it is complicated. My job went part time in January, and I'm contracting on the side. (If anyone needs work done requiring a mix of Perl, databases, math, and ability to talk with business people, my resume is available.) The promise is that I can go back full time in May. We will see what happens.
Anyways one client just turned me on to FreshBooks, which is a great tool for anyone who is contracting. It keeps track of how long you have worked on which projects for which clients. Your clients can see how much you have worked for them, and when. And it will even bill them for you if you want. And it is free if you don't have too many clients.
All in all I am very happy with it right now.
I've known about it for a long time and ignored it because I figured that I knew how to do everything there already. That has been somewhat true - I've solved over half the problems and finding the time to code the solutions has been a bigger challenge than knowing how to do them. But some are challenging. For instance the maximal x for problem 66 has over 30 digits, so you aren't solving it by hand.
I've also learned a bit about Perl. For instance if you have a long list of small integers, it is worth learning about the vec function...
This isn't the journal entry that I intended to write when I first logged into the site last night.
After several experiences of being logged out from use.perl.org very unexpectedly (and often very quickly), my short question is why use.perl.org can't remember for 5 minutes that I am logged in. My follow-up is what alternative site would be recommended that didn't have this problem and had plenty of Perl people.
I hope it remembers that I am logged in for long enough that I can post this.
I finally uploaded statistics-distributions.js to Google Code. In the hope that if someone goes searching for that, they might find it there.
In other news I have become addicted to Stack Overflow. We'll see how long that interest lasts.
Start typing into a form and it creates an autocomplete popup list of things you've typed there in the past. That's nice. But your past typos show up as well. The solution? Navigate to that entry and press shift-delete. Bye bye entry.
I only discovered this by accident. Polling coworkers, most people don't know about it.
What other really useful functionality does Firefox have that nobody knows about?
I got feedback from 12 people, which I'm guessing is about a quarter of the audience. I had a number of people who either didn't know what to expect, or found the math a little hard. The ratings averaged out to 3.67/5. I have no idea how people tend to rate these things. I'm guessing that is at the "improvement needed" end of the scale, which is not entirely surprising considering that it was the first talk that I ever did on this scale. However I'd have liked to do better than that.
Does anyone have any idea how I should be scaling my expectations?