Sthyo

Workaround for "certificate verify failed" and "undefined method 'read_nonblock'"

2012-01-04T12:34:00.004-05:00

If you are getting the error "NoMethodError: undefined method `read_nonblock' for #<OpenSSL::SSL::SSLSocket:...>" or "OpenSSL::SSL::SSLError: certificate verify failed" using JRuby 1.6.5/Ruby 1.9, update JRuby to 1.6.6 or later for the former error, or check out the following.

Thanks to Jeremy for finding the related monkey patch BufferedIO monkeypatch that works in JRuby 1.6.5/Ruby 1.9:


require 'net/http'
require 'net/https'
require 'uri'
require 'openssl'

# Avoid error: NoMethodError: undefined method `read_nonblock' for #<OpenSSL::SSL::SSLSocket:...>
# See: http://jira.codehaus.org/browse/JRUBY-5529
Net::BufferedIO.class_eval do
    BUFSIZE = 1024 * 16
 
    def rbuf_fill
      timeout(@read_timeout) {
        @rbuf << @io.sysread(BUFSIZE)
      }
    end
end

uri = URI.parse('https://host.with.bad.cert/some/path?a=1')
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
# Avoid error, if unavoidable: "OpenSSL::SSL::SSLError: certificate verify failed"
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
request = Net::HTTP::Get.new(uri.request_uri)
response = http.request(request)
puts response.body

Note: The "read_nonblock" error is fixed in JRuby 1.6.6.

All Mobile user-agent Strings in a Text File

2011-11-16T14:44:00.001-05:00

WURFL is a great project dedicated to collecting data on wireless/mobile devices. If you just want a list of the user-agent strings, though, it is quite a bit of XML to sift through. You could parse it, or just convert it to a text file list of mobile user-agents. If you are doing the latter, you could use this script I wrote:


#!/bin/bash

if [ $# -eq 0 ]
then
  echo "Download and extract the latest wurfl.xml from http://sourceforge.net/projects/wurfl/files/WURFL/"
  echo "then use this script to extract the wireless user agent strings."
  echo ""
  echo "usage: wurfltxt wurfl.xml"
  exit 1
fi

if [ ! -f "$1" ]
then
  echo "the first argument must be a valid path to wurfl.xml"
  exit 1
fi

grep user_agent $1 > wurfl.txt
sed 's/.*user_agent="//g' wurfl.txt > wurfl2.txt
sed 's/"\ fall_back.*//g' wurfl2.txt > wurfl.txt
rm wurfl2.txt
sort wurfl.txt > wurfl_sorted.txt
rm wurfl.txt
mv wurfl_sorted.txt wurfl.txt
echo Created wurfl.txt

Save that to a file called wurfltxt and then:


chmod +x wurfltxt
wurfltxt /path/to/wurfl.xml

and it will generate a wurfl.txt file containing just the complete WURFL list of mobile device user-agents.

Tomcat 6 - SEVERE: Error listenerStart - How to Log the Stacktrace

2011-10-14T10:45:00.001-04:00

If you run into the Tomcat 6 error "SEVERE: Error listenerStart", a quick way to get logging to work so you can see the error/exception/stacktrace is to add the following file to a file logging.properties that you should put into webapps/your_webapp_name/WEB-INF/classes (putting it into conf/logging.properties didn't work):


org.apache.catalina.core.ContainerBase.[Catalina].level = FINEST
org.apache.catalina.core.ContainerBase.[Catalina].handlers = java.util.logging.ConsoleHandler

Thanks to Venkatt Guhesan's post for this info.

In our case, the only file in work/Catalina/localhost/your_webapp_name/ was SESSIONS.ser. This seemed to indicate that maybe it was unable to compile something or to load the classes. It turned out to be a missing class/jar.

The resulting stacktrace in my case was similar to:


SEVERE: Exception sending context initialized event to listener instance of class org.springframework.web.context.ContextLoaderListener
java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory
 at org.springframework.web.context.ContextLoader.(ContextLoader.java:139)
 at org.springframework.web.context.ContextLoaderListener.createContextLoader(ContextLoaderListener.java:57)
 at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:48)
 at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3972)
 at org.apache.catalina.core.StandardContext.start(StandardContext.java:4467)
 at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791)
 at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771)
 at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:526)
 at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:905)
 at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:740)
 at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:500)
 at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1277)
 at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:321)
 at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)
 at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
 at org.apache.catalina.core.StandardHost.start(StandardHost.java:722)
 at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
 at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
 at org.apache.catalina.core.StandardService.start(StandardService.java:516)
 at org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
 at org.apache.catalina.startup.Catalina.start(Catalina.java:593)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
 at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.logging.LogFactory
 at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1484)
 at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1329)
 ... 27 more
Oct 14, 2011 10:11:30 AM org.apache.catalina.core.StandardContext start
SEVERE: Error listenerStart
Oct 14, 2011 10:11:30 AM org.apache.catalina.core.StandardContext start
SEVERE: Context [/your_webapp_name] startup failed due to previous errors

Of course, your error is probably different.

Migrating Rails from MySQL to Oracle

2011-09-29T16:03:00.002-04:00

Assuming you have MySQL, Oracle, and Oracle Client (with sqlplus) setup and have already created the schema, then you just do the following: (We created the schema via Rails, first with rake db:schema:dump then rake db:schema:load after adding names to indexes and shortening column names and reference to those attributes. The benefit of doing it that way was to ensure the correct datatypes for Active Record would be created.)


mysqldump -u your_username --password='your_password'  name_of_mysql_database --compatible=oracle --no-create-info --no-create-db --skip-opt --skip-set-charset --skip-extended-insert --skip-comments --skip-quote-names --compact > /some/path/backup.sql
cd /some/path/
grep ដ backup.sql
# Look at the output of grep. If it returns anything, change the unicode character here and later.
# Changes 2 sequential backslashes to a unicode character temporarily.
sed -e "s/\\\\\\\\/ដ/g" backup.sql > backup.sql.1
# Unescape characters that mysqldump escapes.
# Note: there is a good chance that the escape character replacements will fail to work when executed. Because we had a table with heavy use of escape characters that
# made sqlplus choke. You might want to look at the generated SQL and adjust these accordingly. Possibly characters escaped at beginning or end of value or next to each other won't work without more replacements to make those valid, since double single-quote is an escaped single-quote in SQL.
sed -e "s/\\\\'/''/g" \
    -e "s/\\\\0/'||Chr(0)||'/g" \
    -e "s/\\\\a/'||Chr(7)||'/g" \
    -e "s/\\\\b/'||Chr(8)||'/g" \
    -e "s/\\\\t/'||Chr(9)||'/g" \
    -e "s/\\\\n/'||Chr(10)||'/g" \
    -e "s/\\\\f/'||Chr(12)||'/g" \
    -e "s/\\\\r/'||Chr(13)||'/g" \
    -e "s/\\\\e/'||Chr(27)||'/g" \
    -e "s/\\\\"/"/g" \
    backup.sql.1 > backup.sql.2
# The first expression changes the unicode char back to 2 sequential backslashes. The second two expressions fix dates. You may need to alter the date format.
sed -e "s/ដ/\\\\/g" \
    -e "s/'[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9]'/to_date(&,'yyyy-mm-dd HH24:MI:SS')/g" \
    -e "s/'[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]'/to_date(&,'yyyy-mm-dd')/g" \
    backup.sql.2 > backup.sql.3
sqlplus oracle_username/oracle_password@host:port/SID

The following is done in sqlplus:


-- this makes ampersands ok
SET SCAN OFF
-- this commits after every insert
SET AUTOCOMMIT ON
-- this outputs the insert, so that when there is an error, you know what it was trying to execute
SET ECHO ON
-- this eliminates output stating how many rows were created
SET FEEDBACK OFF
-- this shows the current time next to each statement execution, so in the end you know how long it took and when there were errors
SET TIME ON
@/some/path/backup.sql.3

Basically, this does the export, does text alterations to unescape and Oracle SQL-fy, and makes the datetimes and dates into Oracle datetimes and dates. Then it imports it. This may not work for you, or might require adjustments to work properly.

Note: we first tried Oracle SQLDeveloper to migrate the data. It required dba access for the schema user, created two users, granted one of them various rights, and failed in the end. Maybe it would work for you, though.

When you're done, to test out that the data is the same between the MySQL and Oracle you could try using my schema-dump utility to output the data in both databases and diff the two files.

Pentaho was mentioned as a possibility for a migration tool in a Stack Overflow thread about MySQL to PostgreSQL migration, so you might look into that also.

Bash Script to Automate Import of External Cert Given Host and Port Into Java Keystore

2011-09-14T13:03:00.000-04:00

Here's a bash script to import an external cert and import it into the local JDK's keystore, using some commands I found here:

Save this to a file called add_cert, then chmod +x add_cert and put it in your path:


#!/bin/bash

# add_cert
# Written by Gary S. Weaver 2011-09-14
# Downloads cert and imports it automatically into Java keystore.
# This is useful to run immediately after updating the JDK with a new keystore.

# References:
# http://serverfault.com/questions/129503/save-remote-ssl-certificate-via-linux-command-line
# http://www.sslshopper.com/article-most-common-java-keytool-keystore-commands.html
# http://docs.codehaus.org/display/JETTY/How+to+configure+SSL

set -e

KEYTOOL_PATHNAME=$1
KEYSTORE_PATHNAME=$2
KEYSTORE_PASSPHRASE=$3
HOST=$4
PORT=$5

if [ $# -eq 0 ]
then
  echo ""
  echo "usage: add_cert keytool_pathname keystore_pathname passphrase host [optional_port]"
  exit 1
fi

if [ ! -f "$KEYTOOL_PATHNAME" ]
then
  echo "the first argument must be a valid path to the keytool executable (e.g. '\$JAVA_HOME/bin/keytool'): '$KEYTOOL_PATHNAME'"
  echo ""
  echo "usage: add_cert keytool_pathname keystore_pathname passphrase host [optional_port]"
  exit 1
fi

if [ ! -f "$KEYSTORE_PATHNAME" ]
then
  echo "the second argument must be a valid path to the keystore (e.g. '\$JAVA_HOME/jre/lib/security/jssecacerts'): '$KEYSTORE_PATHNAME'"
  echo ""
  echo "usage: add_cert keytool_pathname keystore_pathname passphrase host [optional_port]"
  exit 1
fi

if [ -z "$KEYSTORE_PASSPHRASE" ]
then
  echo "the third argument must be a valid passphrase (e.g. 'changeit'): '$KEYSTORE_PASSPHRASE'"
  echo ""
  echo "usage: add_cert keytool_pathname keystore_pathname passphrase host [optional_port]"
  exit 1
fi

if [ -z "$HOST" ]
then
  echo "the fourth argument must be a valid hostname (e.g. www.acme.org): '$HOST'"
  echo ""
  echo "usage: add_cert keytool_pathname keystore_pathname passphrase host [optional_port]"
  exit 1
fi

if [ -z "$PORT" ]
then
  PORT=443
fi

echo "openssl s_client -connect $HOST:$PORT /tmp/$HOST.pem"
openssl s_client -connect $HOST:$PORT /tmp/$HOST.pem
echo "$KEYTOOL_PATHNAME -keystore jssecacerts -import -alias $HOST -file /tmp/$HOST.pem -trustcacerts -noprompt -storepass $KEYSTORE_PASSPHRASE"
$KEYTOOL_PATHNAME -keystore $KEYSTORE_PATHNAME -import -alias $HOST -file /tmp/$HOST.pem -trustcacerts -noprompt -storepass $KEYSTORE_PASSPHRASE

In addition, here are some helpful commands:

List all certs by alias in alphanumeric order:


$JAVA_HOME/bin/keytool -list -v -keystore $JAVA_HOME/jre/lib/security/keystore_name -storepass store_password | grep "Alias name" | sort

Remove a cert given its alias:


$JAVA_HOME/bin/keytool -delete -alias alias_name -keystore $JAVA_HOME/jre/lib/security/keystore_name -storepass store_password

OCIError: ORA-00001: unique constraint (APP.SYS_C9876543210) violated: INSERT INTO sessions

2011-08-26T12:02:00.004-04:00

Maintaining an older app that uses Rails session persistence in the database SESSIONS table, and after a restore, we got the following issue:

OCIError: ORA-00001: unique constraint (APP.SYS_C9876543210) violated: INSERT INTO sessions (data, created_at, updated_at, session_id, id) VALUES(empty_clob(), TO_DATE('2011-08-26 10:55:08','YYYY-MM-DD HH24:MI:SS'), TO_DATE('2011-08-26 10:55:08','YYYY-MM-DD HH24:MI:SS'), 'd33f1cb3386333e7fa7a4c8b206a0716a08e396f07b28171bd9ed13a14bc8', 3478952) ...

The issue is that the session id that it is trying to insert is not unique, so it is violating the primary key constraint on the table. You can fix this manually by clearing out the SESSIONS table, or you can use:


$ rake db:sessions:clear

Why this happened: in the example above, the '3478952' value is from the sequence associated with the SESSIONS table. I had erroneously requested a restore of tables to the database, so the DBA didn't restore the sequences and other objects with the Oracle user. The restored data (copied from a different environment) had more data in SESSIONS and a greater sequence number. To ensure there won't be other issues with sequences, etc. I'm asking the DBA to do another restore, this time copying all objects.

(I replaced a few values in the error message to make it more generic for this post.)

Dump Database or Schema to a Diffable Text File with Schema Dump

2011-08-26T11:42:00.001-04:00

Just wrote a new tool called Schema Dump that dumps a database or schema data to a text file that you can then use the sort and diff (compare) commands with to do a diff of changes between two schemas or databases. Pretty simple script using Ruby DBI and trollop.

Some examples of schemadump command-line:


$ schemadump -r DBI:OCI8://db.acme.org:1234/ACMETEST -u jdoe -p secret

SAMPLE_TABLE_A: '123', 'Joe'
SAMPLE_TABLE_B: '123', '234'
SAMPLE_TABLE_B: '123', '235'
SAMPLE_TABLE_C: '234', 'Kia', 'Sorento', '345'
SAMPLE_TABLE_C: '235', 'Kia', 'Sportage', '346'
SAMPLE_TABLE_D: '345', 'Red'
SAMPLE_TABLE_D: '346', 'Green'


$ schemadump -r DBI:OCI8://db.acme.org:1234/ACMETEST -u jdoe -p secret > test
$ sort test > test-sorted
$ schemadump -r DBI:OCI8://db.acme.org:3456/ACMEPROD -u jdoe -p secret > prod
$ sort prod > prod-sorted
$ diff test-sorted prod-sorted

Tomcat and Webapp Hot-deploy Permgen "Leaks"

2011-05-10T15:23:00.014-04:00

Spent the last few days trying to fix an issue of running out of permgen when hot-deploying (removing old war to undeploy and adding new war to deploy while Tomcat is running).

Using (old) Tomcat 5.5.20, I attached via JMX remotely from JConsole and watched the permgen climb during deploy, climb a tiny bit during undeploy, then climb more during deploy, etc. as I repeated until running out of permgen for a portlet war (webapp).

I tweaked the settings until I had this in catalina.sh (this is insecurely opening up JMX btw):


JAVA_OPTS="$JAVA_OPTS -Djava.rmi.server.hostname=name.of.the.host \
                      -XX:+UseParNewGC \
                      -XX:+UseConcMarkSweepGC \
                      -XX:+CMSPermGenSweepingEnabled \
                      -XX:+CMSClassUnloadingEnabled \
                      -Dcom.sun.management.jmxremote \
                      -Dcom.sun.management.jmxremote.ssl=false \
                      -Dcom.sun.management.jmxremote.port=(specify open port here) \
                      -Dcom.sun.management.jmxremote.authenticate=false \
                      -Djava.library.path=/usr/lib:/usr/lib64 \
                      -Xmx1024m \
                      -Xms1024m \
                      -XX:MaxPermSize=128m \
                      -XX:NewSize=128m \
                      -XX:MaxTenuringThreshold=0 \
                      -XX:+UseTLAB \
                      -XX:SurvivorRatio=128 \
                      "

(notice: I would have set permgen size to 256m here, but was trying to test it failing due to lack of permgen during hot-deploys.)

I then switched to Tomcat 7.0.12 and did same and also tried out VisualVM. I could see that the permgen even with the same settings started off much lower. The deploys made permgen jump even more than before, though- at least that is how it appeared; I didn't do quantitative measurement even though I probably should have; I was just "spiking" it. The undeploys between the deploys didn't make permgen go up really so that was nice. Long story short: it still ran out of permgen for the portlet (webapp) that I was redeploying.

I tried against and used Tomcat 7's new /manager/text/findleaks. All this did was just print out that the portlet/webapp I was hot-deploying was causing a memory leak. This would have been more useful if I hadn't already been focusing on a particular webapp I knew had an issue. It would really be nice to have something that could tell you not only what webapps were a problem, but could help you diagnose what classes/instances are sticking around after you undeploy that maybe shouldn't be there. I also think that logging leaks may be more helpful than using part of manager, because many don't deploy manager (they remove it).

I then used jmap to create hprofs (dumps) that I first looked at in VisualVM to compare side-by-side. On the server, I did:


ps auxf|grep java

I got the process id and then did the following, first after the first hot-deploy, second after undeploying, and third after redeploying again (just moving around the same war in and out of webapps directory):


jmap /tmp/(some_filename).hprof (process id)

(note: be sure to tar.gz these via tar -czvf filename.hprof before scp'ing them if it helps to save time.)

I tried Mat (Eclipse Memory Analyzer, which is a standalone tool so don't have to have/use Eclipse) to look at the pie charts and dominator trees. I found that after opening/loading the heaps to view side-by-side, when I clicked on Histogram icon (looks like a bar graph) that it would show an icon to the right called "Compare to Another Heap Dump", so I tried that. When I compared the first deploy to the heap after the second deploy, things that stood out (in addition to normal types and collections) were instances resulting from reflection (Method objects), JarFile and URLJarFile instances, and xerces's XSSimpleTypeDecl. However, focusing on the new instances wasn't helpful. I was actually interested in what wasn't new. It would be nice if there were a way to focus on what instances where the same between the two dumps. Is there an easy way of doing that with VisualVM? I didn't see one. And that would require me to eliminate much of the stuff that should stay the same from the results (Tomcat jars, etc.). It just isn't that easy yet.

I also used JHat to browse and crawl the difference:


jhat -J-Xmx2048m -baseline ~/Desktop/after_1st_hotdeploy.hprof ~/Desktop/after_2nd_hotdeploy.hprof

(note: ironically it first ran out of memory when I tried to compare without using -J to give it more memory.)

It was a little harder to make use of browsing the results via served html and showing both new and old instances in the diff. It is also fairly slow and drags the system down a bit. Here at first I was again probably wrongly focused on the new instances rather than the ones that remained between deploys.

Going back and looking at both Mat and JHat later, I'm thinking that I should have removed all but the absolute necessities for the webapp, because all of the classes loaded by Tomcat for every other webapp, common dependencies, etc. get in the way of trying to diagnose what shouldn't have been left behind. And, maybe I should have only focused on what remained after the undeploy.

I read this post/interview about Tomcat 7's memory leak protection and about the various things in webapps that can cause leaks:
http://java.dzone.com/articles/memory-leak-protection-tomcat

It was an older interview and not totally surprising, but it is nice to know that at least someone was working on trying to clean up other people's messes when it comes to memory leaks related to hot-deploys and undeploys.

All that said, here are some recommendations for diagnosing permgen memory leaks related to undeploys/hot-deploys:

Determine which webapps have leaks by either using JConsole or VisualVM to watch permgen as you undeploy and redeploy, or by undeploying and redeploying each webapp and then using Tomcat 7's findleaks to tell you which webapps are leaking.
After determining which webapps to focus on, create a new Tomcat environment like the old one but where you really limit jars and webapps to only those absolutely necessary to test the hot-deploy leak issue.
Finally, do diffs with JHat and determine what is the same (not marked as new) and try to exclude in your mind those packages loaded by Tomcat to focus on the others, or you may use Mat and visually compare dominator trees, etc.

Unfortunately, the last item is the tough one. You might be better off researching possible leaks in versions of jars/libraries you are using and then carefully updating Tomcat and jars used by your webapps to the latest versions- hoping that the leaks have been fixed. Also, do what you can to ensure that threads, connections, and anything else your webapp starts/uses/references (directly or indirectly) is cleaned up as much as possible.

Edit: Christopher Schultz on the Tomcat users mailing list said:

Generally speaking, when you have a permgen "leak" across webapp redeployments, this is the situation:
1. Webapp invokes some code that stores a reference to the webapp's WebappClassLoader
2. That reference is not cleared during undeploy [this is the leak]
The big problem is that the WebappClassLoader keeps a reference to all java.lang.Class objects it loads, so it can be quite bulky when it leaks, depending on exactly how many classes your webapp has locally.
markt has a great presentation which lays out the anatomy of these types of memory leaks, how to diagnose them, and often how to fix them. These are slides from a presentation at last year's ApacheCon NA. It's /very/ readable.
http://people.apache.org/~markt/presentations/2010-11-04-Memory-Leaks-60mins.pdf
- -chris

That is a very helpful presentation!

Useful *nix One-liners

2011-05-06T10:28:00.005-04:00

Just got a few notifications about new answers to the question, "What are the most useful "swiss army knife" one-liners on Unix?" in Quora. Might want to follow it.

Fixing the parseDouble Bug in Java via Oracle Patch

2011-03-25T15:03:00.004-04:00

To fix this bug assuming you have one of the JVMs that the patch will fix, do the following at your own risk:

1. Download fpupdater.zip from http://www.oracle.com/technetwork/java/javase/downloads/index.html#fpupdater

2. Unzip it and copy the fpupdater.jar file contained within the zipped directory to a location on server in a directory you have access to (e.g. /tmp).

3. Determine the user that owns the Java install, and sudo to that user or have your admin do that.

4. Change to the directory that you copied the file to (e.g. /tmp).

5. Test for the bug. (Note: Wherever I just have "java" listed, you may need to call as "/path/to/java/bin/java" if it isn't already in your command path.)


java -jar fpupdater.jar -t -v

If your JVM has the bug, this will likely produce the result: "Verification test failed; timed out."

If your JVM does not have the bug, this will likely produce the result: "Verification test passed." If it says this, there is no need to continue- in fact, you probably shouldn't.

If the command fails with something like:


Warning: -jar not understood. Ignoring.
Exception in thread "main" java.lang.NoClassDefFoundError: fpupdater.jar
   at _ZN4java4lang11VMThrowable16fillInStackTraceEPNS0_9ThrowableE (/usr/lib64/libgcj.so.5.0.0)
   at _ZN4java4lang9Throwable16fillInStackTraceEv (/usr/lib64/libgcj.so.5.0.0)
   at _ZN4java4lang9ThrowableC1EPNS0_6StringE (/usr/lib64/libgcj.so.5.0.0)
   at _ZN4java4lang5ErrorC1EPNS0_6StringE (/usr/lib64/libgcj.so.5.0.0)
   at _ZN4java4lang12LinkageErrorC1EPNS0_6StringE (/usr/lib64/libgcj.so.5.0.0)
   at _ZN4java4lang20NoClassDefFoundErrorC1EPNS0_6StringE (/usr/lib64/libgcj.so.5.0.0)
   at _ZN3gnu3gcj7runtime11FirstThread3runEv (/usr/lib64/libgcj.so.5.0.0)
   at _Z13_Jv_ThreadRunPN4java4lang6ThreadE (/usr/lib64/libgcj.so.5.0.0)
   at _Z11_Jv_RunMainPN4java4lang5ClassEPKciPS4_b (/usr/lib64/libgcj.so.5.0.0)
   at __gcj_personality_v0 (/tmp/java.version=1.4.2)
   at __libc_start_main (/lib64/tls/libc-2.3.4.so)
   at _Jv_RegisterClasses (/tmp/java.version=1.4.2)

that probably means that the Java you are using is gcj/GCC Java and not Sun/Oracle Java. Assuming you have a Sun/Oracle Java JVM to update, you may need to call java from the Java location you are trying to update, e.g.:


/path/to/java/bin/java -jar fpupdater.jar -t -v

6. Patch it (and if you have to, prepend the path to java, see above.)


java -jar fpupdater.jar -u -v

It tells you where it backed the old file up to if you use -v, just in case it doesn't work. Be sure to record that somewhere in case you need it.

7. Test it again (and if you have to, prepend the path to java, see above.)


java -jar fpupdater.jar -t -v

This should result in "Verification test passed."

8. This is the part you need to handle and I can't really help a lot. You should restart everything, if needed, that uses that version of Java. You could try listing processes that use Java to assist with this (however you do it in your environment):


ps auxf|grep java

But, after that you are on your own about how you want to do that.

9. Test everything!

Good luck.

Recovering Lost Java 1.5.0 for Snow Leopard After Java for Mac OS X 10.6 Update 4 Install

2011-03-16T16:00:00.001-04:00

It looks like Java for Mac OS X 10.6 Update 4 just deleted my /System/Library/Frameworks/JavaVM.framework/Versions/1.5.0-leopard directory that I had setup. It is as if Apple assumes that it is ok to delete whatever the /System/Library/Frameworks/JavaVM.framework/Versions/1.5.0 or /System/Library/Frameworks/JavaVM.framework/Versions/1.5 symlinks were pointed to.

To get them back:


sudo port install wget
cd /tmp/
wget http://tedwise.com/files/java.1.5.0-leopard.tar.gz
tar -zxvf java.1.5.0-leopard.tar.gz
sudo mv 1.5.0 /System/Library/Frameworks/JavaVM.framework/Versions/1.5.0-leopard
cd /System/Library/Frameworks/JavaVM.framework/Versions/
sudo rm 1.5.0
sudo ln -s 1.5.0-leopard 1.5.0
sudo ln -s 1.5.0-leopard 1.5

To get Java 1.4, which it doesn't mess with, but just for completeness:


cd /tmp/
wget http://tedwise.com/files/java.1.4.2-leopard.tar.gz
tar -zxvf java.1.4.2-leopard.tar.gz
sudo mv 1.4.2 /System/Library/Frameworks/JavaVM.framework/Versions/1.4.2-leopard
cd /System/Library/Frameworks/JavaVM.framework/Versions/
sudo ln -s 1.4.2-leopard 1.4.2
sudo ln -s 1.4.2-leopard 1.4

This information was originally posted and the binaries hosted by Ted Wise as "Using Java 1.5 and Java 1.4 on Snow Leopard". Thanks again, Ted!

Requesting a Specific Non-primary Virtual Host via HTTPS Behind Shibboleth

2011-03-15T17:03:00.012-04:00

To hit a webapp on specific server on a non-default virtual host on non-default SSL port that is load-balanced and behind Shibboleth (or another SSO), you have a go through a few hoops, if you want the normally load-balanced host/port to resolve to that specific host on the specific server, bypassing the load balancer.

Setup a SSH tunnel with privileged port 443 locally going to the SSL port of the server for the virtual host you are trying to visit:


sudo ssh -L 443:third-foobar-server.acme.org:1443 myuser@third-foobar-server.acme.org

Then I edited /etc/hosts and pointed foobar.acme.org at 127.0.0.1 by adding the entry:


127.0.0.1 foobar.acme.org

Then you can hit https://foobar.acme.org/ in your browser and bypass the load balancer.

This worked in OS X, should work work Linux, and with Putty and the Windows-specific hosts file, it should work there also.

In addition, Scott Cantor suggested to me that webisoget's map command might be a good way of scripting access to individual shibbolized apps on specific servers more simply, for monitoring, etc.

How to Show a Facebook Home News Stream, Friend Requests, and Notifications via Javascript

2011-03-10T17:38:00.006-05:00

This is a mixture of a number of things I found (and didn't find) out there, although the most helpful was Graph api & javascript base Facebook Connect tutorial, so it is somewhat originally based on that.

This is what it looks like:


<div id="fb-root">
</div>
<div id="fb_message">
</div>
<!-- Parts of this are based on http://thinkdiff.net/facebook/graph-api-javascript-base-facebook-connect-tutorial/ -->
<script type="text/javascript">

    var application_id = 'replace with your Facebook application id';
    var application_name = 'replace with your Facebook application name';
    
    if (!(application_id)) {
        document.getElementById('fb_message').style.display = "block";
        document.getElementById('fb_message').innerHTML = "Facebook application_id not defined.";
    } else if (!(application_name)) {
        document.getElementById('fb_message').style.display = "block";
        document.getElementById('fb_message').innerHTML = "Facebook application_name not defined.";
    } else {
        document.getElementById('fb_message').innerHTML = "";
        document.getElementById('fb_message').style.display = "none";
    }
    
    var news_results;
    var friends_results;
    var notification_results;
    var friend_request_results;

    window.fbAsyncInit = function() {
        FB.init({
            appId: application_id,
            status: true,
            cookie: true,
            xfbml: true
        });

        /* All the events registered */
        FB.Event.subscribe('auth.login', function(response) {
            // reload the page so news will show
            window.location.reload();
        });
        FB.Event.subscribe('auth.logout', function(response) {
            document.getElementById('fb_status').style.display = "none";
            document.getElementById('fb_news').style.display = "none";
            document.getElementById('fb_dashboard').style.display = "none";
        });

        FB.getLoginStatus(function(response) {
            if (response.session) {
                document.getElementById('fb_news').style.display = "block";
                document.getElementById('fb_news').innerHTML = "Asking Facebook for latest news from your friends...";
                var q1 = FB.Data.query("SELECT created_time, actor_id, message, permalink FROM stream WHERE source_id in (SELECT target_id FROM connection WHERE source_id=me() AND is_following=1) AND strlen(message) > 0 AND is_hidden = 0 limit 5");
                document.getElementById('fb_news').innerHTML = "Asking Facebook for name and pictures of friends with news...";
                var q2 = FB.Data.query("SELECT uid, pic_square, name FROM user WHERE uid IN (SELECT actor_id FROM stream WHERE source_id in (SELECT target_id FROM connection WHERE source_id=me() AND is_following=1) AND strlen(message) > 0 AND is_hidden = 0 limit 5)");
                document.getElementById('fb_news').innerHTML = "Asking Facebook for number of notifications...";
                var q3 = FB.Data.query("SELECT notification_id FROM notification WHERE recipient_id=me() AND is_hidden = 0");
                document.getElementById('fb_news').innerHTML = "Asking Facebook for number of friend requests...";
                var q4 = FB.Data.query("SELECT uid_from FROM friend_request WHERE uid_to=me()");
                document.getElementById('fb_news').innerHTML = "Waiting on Facebook...";
                FB.Data.waitOn([q1, q2, q3, q4], function(args) {
                    document.getElementById('fb_news').innerHTML = "Processing Facebook data...";
                    news_results = args[0];
                    friends_results = args[1];
                    notification_results = args[2];
                    friend_request_results = args[3];
                    displayFacebookData();
                    document.getElementById('fb_dashboard').style.display = "block";
                    document.getElementById('fb_status').style.display = "block";
                });
            }
        });

        FB.XFBML.parse();
    };
    (function() {
        var e = document.createElement('script');
        e.type = 'text/javascript';
        e.src = document.location.protocol + '//connect.facebook.net/en_US/all.js';
        e.async = true;
        document.getElementById('fb-root').appendChild(e);
    }());

    function displayFacebookData() {
        document.getElementById('fb_dashboard').innerHTML = "<table>\n<tbody>\n<tr>\n";
        document.getElementById('fb_news').innerHTML = "Displaying Facebook notifications...";
        if (notification_results != null) {
            document.getElementById('fb_dashboard').innerHTML += "<td><a href='http://www.facebook.com/'>Notifications: " + notification_results.length + "</a></td>\n";
        }
        else {
            document.getElementById('fb_dashboard').innerHTML += "<td><a href='http://www.facebook.com/'>Notifications: None</a></td>\n";
        }
        document.getElementById('fb_news').innerHTML = "Displaying Facebook friend requests...";

        if (friend_request_results != null) {
            document.getElementById('fb_dashboard').innerHTML += "<td><a href='http://www.facebook.com/friends/edit/?sk=requests'>Friend Requests: " + friend_request_results.length + "</a></td>\n";
        }
        else {
            document.getElementById('fb_dashboard').innerHTML += "<td><a href='http://www.facebook.com/friends/edit/?sk=requests'>Friend Requests: None</a></td>\n";
        }
        document.getElementById('fb_dashboard').innerHTML += "</tr>\n</td>\n</tbody>\n</table>\n";
        document.getElementById('fb_news').innerHTML = "Done preparing Facebook dashboard. Displaying news from friends: " + news_results.length + " news items from " + friends_results.length + " users...";

        if (!(friends_results.length > 0)) {
            document.getElementById('fb_news').innerHTML = 'No news from friends. <a href="#" onclick="window.location.reload();">Click here to refresh.</a>';
        }
        else {
            try {
                var tableHtml = "";
                tableHtml = '\n<table style="width:100%;">\n  <tbody>\n';

                var uidToFriendResult = new Object();
                for (i = 0; i < friends_results.length; i++) {
                    uidToFriendResult[friends_results[i].uid] = friends_results[i];
                }
                for (i = 0; i < news_results.length; i++) {
                    if (i % 2 == 0) {
                        rowClass = 'fb_normal';
                    } else {
                        rowClass = 'fb_alternate';
                    }

                    var friendImageUrl = "img/unknown.gif";
                    var friendName = "Unknown";
                    var friendResult = uidToFriendResult[news_results[i].actor_id];
                    if (friendResult != null) {
                        if (friendResult.pic_square != null) {
                            friendImageUrl = friendResult.pic_square;
                        }

                        if (friendResult.name != null) {
                            friendName = friendResult.name;
                        }
                    }

                    var msg = "";
                    var date = "";
                    var postUrl = "http://www.facebook.com/?ref=home#permalink-for-post-was-not-supplied-by-facebook";
                    if (news_results != null && news_results[i] != null) {
                        if (news_results[i].permalink != null) {
                            postUrl = news_results[i].permalink;
                        }

                        msg = news_results[i].message;
                        if (msg == null) {
                            msg = "";
                        } else if(msg.length > 160) {
                            msg = msg.substring(0,160)+"... (more)";
                        }

                        if (news_results[i].created_time != null && news_results[i].created_time > 0) {
                            var d = new Date();
                            d.setTime(news_results[i].created_time * 1000);
                            var year = d.getFullYear().toString().slice(2);
                            var month = d.getMonth() + 1;
                            if (month < 10) { month = "0" + month; }
                            var date = d.getDate();
                            if (date < 10) { date = "0" + date; }
                            var hour = d.getHours();
                            var ampm = "am";
                            if (hour > 11) { ampm = "pm"; }
                            if (hour > 12) { hour = hour - 12; }
                            if (hour == 0) { hour = 12; }
                            if (hour < 10) { hour = "0" + hour; }
                            var minutes = d.getMinutes();
                            if (minutes < 10) { minutes = "0" + minutes; }
                            date = month + "/" + date + "/" + year + " " + hour + ":" + minutes + ampm;
                        }
                    }
                    tableHtml += "    <tr class='";
                    tableHtml += rowClass;
                    tableHtml += "'>\n      <td><a href='" + postUrl + "' target='_blank'><img src='";
                    tableHtml += friendImageUrl;
                    tableHtml += "' alt='";
                    tableHtml += friendName;
                    tableHtml += "' title='";
                    tableHtml += friendName;
                    tableHtml += "' style='width:auto; height:20px;'/></a></td>\n      <td><a href='" + postUrl + "' target='_blank'>";
                    tableHtml += friendName;
                    tableHtml += "</a></td>\n      <td><a href='" + postUrl + "' target='_blank'>";
                    tableHtml += msg;
                    tableHtml += "</a></td>\n      <td><span style='white-space: nowrap;'><a href='" + postUrl + "' target='_blank'>";
                    tableHtml += date;
                    tableHtml += "</a></span></td></tr>\n";
                }
                tableHtml += '  </tbody>\n</table>\n';

                document.getElementById('fb_news').innerHTML = tableHtml;
            } catch (e) {
                var errorMsg = '';
                for (var property in e) {
                    errorMsg += property + ': ' + e[property] + '\r\n';
                }
                document.getElementById('fb_news').innerHTML = "<pre>" + errorMsg + "</pre>";
            }
        }
    }

    function setStatus() {
        status1 = document.getElementById('fb_status_text').value;
        FB.api({
            method: 'status.set',
            status: status1
        }, function(response) {
            if (response == 0) {
                alert('Your Facebook status was not updated. In your Facebook profile\'s privacy settings, remove ' + my_facebook_application + ' from your privacy settings, go back to this page, login, and agree to give the application the required privacy settings.');
            } else {
                alert('Your Facebook status was updated');
            }
        });
    }
</script>
<p>
    <fb:login-button autologoutlink="true" perms="status_update,publish_stream,read_stream,offline_access">
    </fb:login-button>
</p>
<div id="fb_dashboard">
</div>
<div id="fb_news">
<a href="#" onclick="window.location.reload();">Click here to refresh news</a>
</div>
<div id="fb_status">
<textarea id="fb_status_text" cols="50" rows="2" placeholder="Post status to Facebook">
</textarea>
<br />
<input type="button" value="Post Status to Facebook" onclick="setStatus(); return false;"/>
</div>

Edit: Now part of the Facebook Portlet v2.0.

Java Thread Dumps, JStack, JDA, JConsole, VisualVM

2011-02-16T13:49:00.007-05:00

Eric D. and Chuck Frank on the uPortal list were discussing Java dumps and analysis. Eric suggested that he does at least three Java thread dumps spaced 30 seconds apart and then analyzes them. Between the two they mentioned:

JStack to dump (or kill -3 (pid) on *nix)
TDA for Java thread dump analysis
JConsole for JVM monitoring in Java 5+

To add some others:

VisualVM included in Java 6 update 7 (on OS X was included in Mac OS X 10.5 Update 4)
Open Source Profilers in Java

But, I am guessing that the point of sample dumps vs. using a profiler was due to what was mentioned by Mike Dunlavey here and here. Good reads, btw.

DoubleRenderError and :before_filter

2011-02-08T11:28:00.014-05:00


A ActionController::DoubleRenderError occurred in 
(some action):

Render and/or redirect were called multiple times 
in this action. Please note that you may only call 
render OR redirect, and at most once per action. 
Also note that neither redirect nor render 
terminate execution of the action, so if you want 
to exit an action after redirecting, you need to 
do something like "redirect_to(...) and return".

When you get a DoubleRenderError, there is a good chance that something is happening as part of an action that you didn't anticipate, because you expected all action to stop when you did a render or redirect_to. So, bad things may be happening, like ignoring authentication, executing the action anyway, and then displaying to the user that either that he was unauthorized or displaying a page that is the result of a DoubleRenderError.

One cause of this is particularly insidious. :before_filter keeps track of whether a render or redirect_to was called and, if so, will stop the action. If you move a method previously called by the :before_filter to the action method itself, you may forget to return from the action if that method calls render or redirect_to. If you are lucky there will be another render or redirect_to and this will result in a DoubleRenderError, so you'll catch it and perhaps be alerted that something went wrong. However, an action getting executed unintentionally is not a good thing.

To avoid this, be sure that when you call a method that might render or redirect for that method to return a boolean that you can use in the calling method to return from the action.

The following authenticate method is fine when called by a :before_filter:


class MyController < ApplicationController

  :before_filter authenticate

  def authenticate
    render :file => "public/401.html", :status => :unauthorized unless authorized?
  end

  def some_action
    # do something important and possibly render or redirect_to
  end

end

However, after moving this to the action method, you need to do a return:


class MyController < ApplicationController

  def authenticate
    auth = authorized?
    render :file => "public/401.html", :status => :unauthorized unless auth
    return auth
  end

  def some_action
    return unless authenticate
    # do something important and possibly render or redirect_to
  end

  def some_other_action
    # this action is not authenticated or is authenticated differently,
    # but still makes sense to include in this controller
  end

end

OS X/Linux Command to Display Total Size of All Directories and Subdirectories Within Current Directory

2011-01-21T09:39:00.007-05:00

Jim just taught me this, which rolls up the sizes of subdirectories to provide a total size of each directories in the current directory in *nix:


du -sch *

I'd used 'du -h' before quite a bit which shows sizes throughout directories recursively. With larger directories, it is helpful to sort and axe the human-readable sizes with 'du -sk * | sort -nr'. 'df -H' is also very helpful to show volume sizes.

Styho Blog Post Stats

2011-01-20T17:21:00.002-05:00

Had a few minutes waiting on a process to finish, so used Excel to graph some blog posting stats.

Schema Sleuth Finds Related Data Automatically

2011-01-07T09:52:00.010-05:00

Just wrote Schema Sleuth to help "spider" a database or schema to find all related data, and to output it to the console as text-based representations of partial tables or SQL inserts. In my case, I wanted everything related to a User ID and its related data, but it can find almost any subset of related data easily. It's written to be called as a Ruby command-line/CLI script in *nix. Tested with OS X against a few Oracle schemas.

Easily View Logs, Changesets, and Authors in Git, Mercurial, and Subversion from Terminal.app in OS X

2010-12-21T14:27:00.007-05:00

I like doing most of my SCM/VCS work in Terminal.app, but when I want to look at logs and the diffs in their changesets for commits, I like a UI.

GitX is great at this. If you've installed Git or have it already, install GitX, start it, and in the menu do "GitX -> Enable Terminal Usage". Restart Terminal.app, browse to a Git repository, and type the following to see the log:


gitx

Subversion was a little trickier. Assuming you've installed Subversion or have it already, install SvnX. Start the application once and it will put the svnx shell script into your ~/bin/svnx. Assuming you don't have ~/bin/ on path, you can add the following to your ~/bash_profile:


alias svnx='~/bin/svnx wc . && ~/bin/svnx log .'

After restarting Terminal.app, browse to a checked-out subversion repository and just type the following to see a log:


svnx

Mercurial is a little tricky, but mostly because the tool I use is called Murky, not hgx, as you might think. Hgx is easier to remember, that is what I call my alias. Assuming you've installed subversion or have it already, install Murky. Then, add the following to your ~/bash_profile:


alias hgx='open -a Murky .'

After restarting Terminal.app, browse to a checked-out Mercurial repository and type the following to see its log:

hgx

(Thanks to Jens Alfke for the Mercurial command-line solution.)

Type of Plane and Seats for Airline by Flight #

2010-12-03T10:24:00.007-05:00

Check out SeatGuru. While many airlines have seat maps online that you can view after you've chosen the flight based on date, time, and price, I've often wished that I could have chosen the type of plane also. With SeatGuru, you can. And, you can even find by route. (Thanks to this post on the Atlassian Developer Blog for the reference.)

Automate Terminal.app ~/.login.sh Script on Login in OS X 10.6 to Mount TrueCrypt Volume, etc.

2010-11-11T14:40:00.002-05:00

TrueCrypt is nice, but it doesn't currently have a way in OS X to automatically ask for just your password at login. I first wrote a simple Applescript in AppleScript Editor:


do script "/Applications/TrueCrypt.app/Contents/MacOS/TrueCrypt /path/to/volume/file/volume.tc /path/to/dir/my_encrypted_dir"

then File > Save As... > File Format: Application and saved to the Applications directory. Then I went to System Preferences -> Accounts -> (choose user) -> Login Items and added that new script (application) there, so it would start automatically after Login.

But, this pops up a UI that gives you the option to use password or key files. I only wanted to provide a password. I also thought it would be cool to have a shell script to run on OS X login in Terminal.app that I could keep in my home directory and modify at will with a text editor rather than AppleScript Editor.

So, I wrote the following AppleScript in AppleScript Editor and again used File > Save As... > File Format: Application to the Applications directory. Then went to System Preferences -> Accounts -> (choose user) -> Login Items and added that new script (application) there, so it would start automatically after Login. (I also removed the other one I'd created.)


tell application "System Events"
 set terminal_is_open to (exists process "Terminal")
end tell

tell application "Terminal"
 activate
 if terminal_is_open then
  tell application "System Events" to tell process "Terminal" to keystroke "n" using command down
 end if
 do script with command "~/.login.sh" in selected tab of the front window
end tell

Then I created a file ~/.login.sh containing:


#/usr/sh

# Things to run on login...
clear

# Mount encrypted volume
/Applications/TrueCrypt.app/Contents/MacOS/TrueCrypt -t -k "" --protect-hidden=no /path/to/volume/file/volume.tc /path/to/dir/my_encrypted_dir

Then I did:


chmod +x ~/.login.sh

Logout, login, and it should display a Terminal.app window prompting you for your password to the TrueCrypt volume. After that, I just leave the window open in case there is an error.

Note that to rename the TrueCrypt volume mount on the desktop, just rename it like any other file.

Time Equivalence or Closeness in RSpec

2010-11-04T16:47:00.007-04:00

If you have two times in RSpec that you want to make sure are equal, you could use:


kid.sleepy_at.should eq(bedtime)

But, that only checks to make sure that the time to the nearest second is equal.

To be more exact than a rounded second use:


kid.sleepy_at.to_f.should eq(bedtime.to_f)

If you want to ensure that a time is just close to the expected time, use be_close.

For example, if sleepy_at should be within a half second of bedtime use:


kid.sleepy_at.to_f.should be_close(bedtime.to_f, 0.5)

Converting time to float with to_f doesn't look pretty when it fails, but if you don't use it, and use:


kid.sleepy_at.should be_close(bedtime, 0.5)

then it is comparing seconds, and you might get an error like:


Failure/Error: kid.sleepy_at.should be_close(bedtime, 0.5)
expected Thu Nov 04 20:50:21 UTC 2010 +/- (< 0.5), got 2010-11-04 20:50:22 UTC

This is due to the second being rounded off.

To make sure they are within a rounded second of each other, use:


kid.sleepy_at.should be_close(bedtime, 1)

Resizing Cross-domain Content in an iFrame in YouTube Video Embeds

2010-10-27T11:21:00.003-04:00

Saw the post, "HTML5 Video Available on the Web – October Update" by Frank Sinton, which linked to YouTube's beta support for embedding via iFrame. I noticed that the linked YouTube post, "A New Way To Embed YouTube Videos", was in a different domain than the embedded video. So, I looked a little deeper.

I chose a currently popular Monty Python video clip from Life of Brian, and looked at the HTML (as of 2010-10-27):


<!DOCTYPE html>
<html>
<head>
    <title>YouTube - Monty Python's - Life of Brian-5</title>

  <style>
    body {
      font: 12px Arial, sans-serif;
      background-color: #000000;
      color: #FFFFFF;
      height: 100%;
      width: 100%;
      margin: 0;
      overflow: hidden;
    }
  </style>
    
    <script  src="http://s.ytimg.com/yt/jsbin/www-embed-vflqRiLRQ.js"></script>

</head>
<body>
  <div id="watch-longform-ad" style="display:none;">

    <div id="watch-longform-text">
Advertisement
    </div>
    <div id="watch-longform-ad-placeholder"><img src="http://s.ytimg.com/yt/img/pixel-vfl3z5WfW.gif" height="60" width="300" /></div>
  </div>

    <div id="html5-player-css-holder"></div>
  <div id="embed-holder">
    <div id="watch-player-div" class="flash-player" style="position: absolute; width:100%; height:100%;"></div>
      <div id="video-player" class="html5-video-player" tabindex="0">

    <div class="video-fallback" style="display: none;">
      Your browser does not currently recognize any of the video formats available.<br>
      <a href="/html5">Click here to visit our frequently asked questions about HTML5 video.</a>
    </div>
    <div id="captions" class="video-captions"><div class="captions-holder"><span id="captions-text" class="hidden"></span></div></div>
    <div class="video-content">
      <div class="html5-video-loader html5-center-overlay html5-icon"></div>
        <svg class="html5-big-play-button html5-center-overlay">

          <g opacity="0.4">
            <path fill="none" stroke="#FFFFFF" stroke-width="3" stroke-linecap="round" stroke-linejoin="round" d="M88.34,9.698
              c-1.09-1.04-2.43-1.62-3.92-1.8c-23.061-1.44-46.08-1.44-69.14,0c-1.48,0.18-2.72,0.76-3.84,1.8c-1.09,1.04-1.81,2.27-2.14,3.78
              c-1.05,7.07-1.56,14.17-1.56,21.26c0,7.21,0.51,14.27,1.56,21.34c0.33,1.51,1.05,2.74,2.14,3.779c1.12,1.051,2.36,1.621,3.84,1.801
              c23.06,1.439,46.08,1.439,69.14,0c1.49-0.18,2.83-0.75,3.92-1.801c1.12-1.039,1.77-2.34,2.1-3.819
              C91.46,48.938,92,41.878,92,34.738c0-7.09-0.54-14.16-1.561-21.22C90.109,12.038,89.46,10.818,88.34,9.698z"></path>
          </g>
          <g class="html5-overlay-button-background" opacity="0.8">
            <path d="M88.34,9.698c-1.09-1.04-2.43-1.62-3.92-1.8c-23.061-1.44-46.08-1.44-69.14,0c-1.48,0.18-2.72,0.76-3.84,1.8
              c-1.09,1.04-1.81,2.27-2.14,3.78c-1.05,7.07-1.56,14.17-1.56,21.26c0,7.21,0.51,14.27,1.56,21.34c0.33,1.51,1.05,2.74,2.14,3.779
              c1.12,1.051,2.36,1.621,3.84,1.801c23.06,1.439,46.08,1.439,69.14,0c1.49-0.18,2.83-0.75,3.92-1.801
              c1.12-1.039,1.77-2.34,2.1-3.819C91.46,48.938,92,41.878,92,34.738c0-7.09-0.54-14.16-1.561-21.22
              C90.109,12.038,89.46,10.818,88.34,9.698z"></path>
          </g>
          <path opacity="0.19" fill="#FFFFFF" enable-background="new" d="M88.34,9.698c-1.09-1.04-2.43-1.62-3.92-1.8
            c-23.061-1.44-46.08-1.44-69.14,0c-1.48,0.18-2.72,0.76-3.84,1.8c-1.09,1.04-1.81,2.27-2.14,3.78c-1.05,7.07-1.56,14.17-1.56,21.26
            c0,7.21,1.49,21.949,1.56,21.34c30.89-29.62,44.3-5.48,81.14-42.56C91.51,12.438,89.46,10.818,88.34,9.698z"></path>
          <g class="html5-overlay-button-background" opacity="0.66">
            <polygon fill="#FFFFFF" points="39.32,16.729 39.32,52.798 68.18,34.168"></polygon>

          </g>
        </svg>
      <ul class="html5-context-menu yt-uix-button-menu hid">
        <li>
          <a class="yt-uix-button-menu-item" target="_blank" href="http://www.youtube.com/watch?v=dQw4w9WgXcQ">Save Video As...</a>
        </li>
        <li>
          <a class="yt-uix-button-menu-item" href="/html5">About HTML5</a>

        </li>
      </ul>
    </div>
    <div class="video-controls" style="display: none;">
      <img class="html5-watermark html5-stop-propagation html5-icon" src="http://s.ytimg.com/yt/img/pixel-vfl3z5WfW.gif" alt="watermark">
      <div class="html5-progress-bar html5-stop-propagation">
        <div class="html5-scrubber-button html5-progress-item html5-icon"></div>
        <div class="html5-progress-list html5-progress-item">
          <div class="html5-play-progress html5-progress-section"></div>

          <div class="html5-load-progress html5-progress-section"></div>
        </div>
      </div>
      <div class="html5-player-chrome html5-stop-propagation">
        <div class="html5-play-button html5-button html5-control">
          <input type="image" class="html5-icon" src="http://s.ytimg.com/yt/img/pixel-vfl3z5WfW.gif" alt="Play/Pause">
        </div>
        <div class="html5-volume-control html5-control yt-uix-tooltip" tabindex="0" title="Mute" data-alt-title="Unmute">
          <div class="html5-volume-button html5-button" data-value="loud">

            <input type="image" class="html5-icon" src="http://s.ytimg.com/yt/img/pixel-vfl3z5WfW.gif" tabindex="-1">
          </div>
          <div class="html5-volume-panel">
            <div class="html5-volume-slider html5-icon"></div>
          </div>
        </div>
        <div class="progress-text html5-control">
          <span class="current-time">00:00</span><span> / </span><span class="duration-time">00:00</span>

        </div>

        <div class="html5-fullscreen-button html5-button html5-control html5-control-right yt-uix-tooltip" data-value="fullscreen" title="Full screen" data-alt-title="Exit full screen">
          <input type="image" class="html5-icon" src="http://s.ytimg.com/yt/img/pixel-vfl3z5WfW.gif" alt="Full screen">
        </div>
        <div class="html5-captions-button html5-button-popup-menu html5-control html5-control-right">
          <span class="html5-button-label html5-empty-label">
            <input type="image" class="html5-icon" src="http://s.ytimg.com/yt/img/pixel-vfl3z5WfW.gif" alt="Subtitles">
          </span>

        </div>
        <div class="html5-quality-button html5-button-popup-menu html5-control html5-control-right" title="Quality">
          <span class="html5-button-label">360p</span>
          <ul class="html5-quality-popup-menu html5-popup-menu">
            <li class="html5-popup-menu-item" data-value="1080p">
              1080p <sup>HD</sup>
            </li>

            <li class="html5-popup-menu-item" data-value="720p">
              720p <sup>HD</sup>
            </li>
            <li class="html5-popup-menu-item" data-value="480p">
              480p
            </li>
            <li class="html5-popup-menu-item" data-value="360p">
              360p
            </li>

          </ul>
        </div>
        <div class="html5-speed-button html5-button-popup-menu html5-control html5-control-right" title="Speed">
          <span class="html5-button-label">Normal</span>
          <ul class="html5-speed-popup-menu html5-popup-menu">
            <li class="html5-popup-menu-item" data-value="2.0">
2x Speed
            </li>
            <li class="html5-popup-menu-item" data-value="1.5">

1.5x Speed
            </li>
            <li class="html5-popup-menu-item" data-value="1.0">
Normal Speed
            </li>
            <li class="html5-popup-menu-item" data-value="0.5">
&frac12; Speed
            </li>
            <li class="html5-popup-menu-item" data-value="0.25">
&frac14; Speed
            </li>
          </ul>

        </div>
        <a class="html5-control html5-control-right html5-control-last html5-player-branding" href="/html5">
          HTML5
        </a>
      </div>
    </div>
  </div>

  </div>

  <script type="text/javascript">
  var playerElementId = "";

  var writeFlashEmbed = function() {
    var video_url = "http://www.youtube.com/v/ZNeq2Utm0nU?showinfo=0&amp;enablejsapi=1&amp;version=3&amp;et=OEgsToPDskKqrAsC70ymD-QZmm4A977Q&amp;iurl=http%3A%2F%2Fi3.ytimg.com%2Fvi%2FZNeq2Utm0nU%2Fhqdefault.jpg&amp;el=embedded&amp;use_native_controls=False&amp;video_id=ZNeq2Utm0nU&amp;autohide=0&amp;hl=en_US&amp;eurl=&amp;autoplay=0";
    var fo = new SWFObject(video_url, "movie_player", "100%", "100%", "7", "#000000");
    var startTime = yt.www.watch.player.processLocationHashSeekTime();
    if (window.opener
      && window.opener.yt
      && window.opener.yt.getConfig
      && window.opener.yt.getConfig('SEQUENTIAL_VIDEO_LIST')) {
      fo.addVariable("playlist", window.opener.yt.getConfig('SEQUENTIAL_VIDEO_LIST'));
    }
    fo.addParam("allowFullscreen", "true");
    fo.addParam("AllowScriptAccess", "always");

    if (startTime) {
      fo.addVariable('start', startTime);
    }
    fo.write("watch-player-div");

    playerElementId = "movie_player";

      handleResize = function() {
    var windowHeight = window.innerHeight;
    var adjustedHeight = windowHeight - _gel('watch-longform-ad').offsetHeight;
    var percentHeight = Math.round((adjustedHeight * 100) / windowHeight) + "%";
    _gel('watch-player-div').style.height = percentHeight;
  }
  yt.events.listen(window, 'resize', handleResize);
  yt.events.listen(_gel('watch-longform-ad-placeholder'), 'resize', handleResize);

  };

  var writeHtml5Embed = function(availableFormats) {
    document.getElementById("html5-player-css-holder").innerHTML = "  <link  rel=\"stylesheet\" href=\"http:\/\/s.ytimg.com\/yt\/cssbin\/www-player-vflbyieqp.css\">\n";

    var startTime = yt.www.watch.player.processLocationHashSeekTime();

    var videoPlayer = new yt.player.VideoPlayer();
    videoPlayer.setAvailableFormats(availableFormats);

    var playerVars = {"iurl": "http:\/\/i3.ytimg.com\/vi\/ZNeq2Utm0nU\/hqdefault.jpg", "el": "embedded", "use_native_controls": false, "video_id": "ZNeq2Utm0nU", "autohide": "0", "hl": "en_US", "eurl": "", "autoplay": "0"};
    if (startTime) {
      playerVars['start'] = startTime;
    }
    videoPlayer.setTargetElementId("video-player");
    videoPlayer.setVideoData(playerVars);
    videoPlayer.initialize(startTime, true, true);

    // js api dispatcher
    // Only works now for void fns
    // TODO: figure out how to do callbacks synchronously, and have arguments, probably with JSON

    playerElementId = "video-player";
      handleResize = function() {
    var windowHeight = window.innerHeight;
    var adjustedHeight = windowHeight - _gel('watch-longform-ad').offsetHeight;
    var percentHeight = Math.round((adjustedHeight * 100) / windowHeight) + "%";
    _gel('video-player').style.height = percentHeight;
  }
  yt.events.listen(window, 'resize', handleResize);
  yt.events.listen(_gel('watch-longform-ad-placeholder'), 'resize', handleResize);

  }

  var availableFormats = yt.player.VideoFormat.formatListFromMap(
      []);

  var supportsHtml5 = yt.player.VideoFormat.hasSupportedFormats(availableFormats);

  if (supportsHtml5) {
    writeHtml5Embed(availableFormats);
  } else {
    _gel('embed-holder').removeChild(_gel('video-player'));
    writeFlashEmbed();
  }

  var playerElement = document.getElementById(playerElementId);

  //add JS API for iframe
  var whitelistApiCalls = {
    'playVideo': true,
    'pauseVideo': true,
    'seekTo': true,
    'mute': true,
    'unMute': true};
  var receiveYtMessage = function(event) {
    var message = event.data || (event.J && event.J.data);
    if (typeof(message) == 'string') {
      // Chrome can handle message data being a non-string
      // object, but Firefox can't; best to always use JSON
      message = goog.json.parse(message);
    }
    var fnName = message && message['f'];
    var argList = (message && message['a']) || [];
    if (whitelistApiCalls[fnName] && playerElement[fnName]) {
      var fn = playerElement[fnName];
      var result;
      if (fn) {
        result = fn.apply(playerElement, argList);
      }
    }
    // TODO(libra): return result?

    // TODO: figure out how to do callbacks synchronously
    //event.source.postMessage("1234.567", event.origin);
  };
  yt.events.listen(window, 'message', receiveYtMessage);


  </script>


</body>
</html>

Although the developer(s) looked to be having the same cross-domain communication issues that others have combatted before using anchors (see my previous post- at least I think that is what those comments are about- I've not looked carefully), it is great that there is a neat solution in code for resizing the content within the iFrame (to do full-screen).

Spring Root, Portlet, and Servlet Contexts

2010-10-20T14:31:00.006-04:00

Note: the following pertains to spring-webmvc-portlet in Spring 2.5.6 with JSR-168 compliant portlets. It may be different in other versions.

web.xml defines how the root application context (an instance of XmlWebApplicationContext) should look for configuration to setup beans in its context. If you define one or more files in contextConfigLocation, it loads those to define the beans, etc. for the root application context, to be available for the entire webapp. If singleton scope (the default for beans), they will be instantiated and configured on webapp load.

For example, web.xml in the Jasig calendar portlet:

/WEB-INF/web.xml

defines the following as config files to use for the root application context via contextConfigLocation:

portlet.xml defines how the portlet application context (an instance of XmlPortletApplicationContext) should look for configuration to setup beans in its context. If you define one or more files in contextConfigLocation, it loads those to define the beans, etc. for portlet application context, to be available for the entire webapp. If singleton scope (the default for beans), they will be instantiated and configured on portlet load, which does not happen until the first time the portlet is loaded (the first request to a page that contains that portlet).

For example, portlet.xml in the calendar portlet:

/WEB-INF/portlet.xml

defines the following as config files to use for the portlet application context via contextConfigLocation:

/WEB-INF/context/portlet/calendar.xml

web.xml also can define one or more additional servlets (e.g. for Ajax functionality). If the servlet-class is org.springframework.web.servlet.DispatcherServlet, it will look for (servlet-name)-servlet.xml as the spring config file that defines the beans for the servlet application context. I don't know for sure, but I'm guessing if singleton scope (the default for beans), they will be instantiated and configured on portlet load, which does not happen until the first time the portlet is loaded (the first request to a page that contains that portlet), unless the servlet is configured to load-on-startup in web.xml.

For example, web.xml in the calendar portlet:

/WEB-INF/web.xml

defines a servlet with servlet-name "spring" as a org.springframework.web.servlet.DispatcherServlet with load-on-startup, so it will load (servlet-name)-servlet.xml which is spring-servlet.xml:

/WEB-INF/spring-servlet.xml

Defining the same context files both in web.xml and portlet.xml is usually wrong, because it basically means you are creating instances of beans in the root *and* portlet contexts. The portlet would always use the ones in the portlet contexts, so the root ones would be wasted, along with any resources they consumed.

Thanks to Eric Dalquist for pointing me in the right direction for this!

Nerot - Flexible Java Scheduling and Caching

2010-10-06T19:24:00.003-04:00

Just did the initial import of Nerot to Github. If you're looking for a fast and flexible Java scheduling library that is easily extensible and that has built-in support for RSS retrieval, calling any method (static or instance) on any Java object, in-memory caching, then check it out!

It's primary immediate purpose for me to make keeping RSS feeds up-to-date (which it grabs via Rome) without even having to wait on retrieval possible. This is done by scheduling (via Quartz, Spring, and dynamic Spring bean creation) update of the feed and always retrieving the feed that is cached in-memory and never making the client wait when they attempt to get the feed.

This also means that it is an unreliable caching mechanism, i.e. if it has not been retrieved before in-memory, then it won't be there when you ask for it.

At first this might seem to have limited application, but, if you think about it, most of the time an almost immediate response is better than waiting for a response because the cache expired, so why should the cache ever expire? So, it doesn't, and you never have to wait for a refresh since it is done in the background.

Workaround for "ORA-04043: object * does not exist" in rake db:migrate

2010-09-28T15:47:00.000-04:00

This has happened at least twice now, so I thought I'd share a workaround.

If you do a production migration in Rails:


RAILS_ENV=production rake db:migrate

and it dies with:


(in /path/to/some/project)
rake aborted!
ORA-04043: object plural_name_of_a_model does not exist
/path/to/some/project/Rakefile:10
(See full trace by running task with --trace)

Then, first try using the --trace option:


RAILS_ENV=production rake db:migrate --trace

If the error is coming from a controller, move that controller and any others that fail to the base project directory (or somewhere outside of where it is supposed to be) and then run the migration again.

Be sure to move the controllers back afterwards.

Goat-time

2010-09-23T15:25:00.005-04:00

Text:

松本武
View profile Translate to Translated (View Original)
(4 users) More options Sep 21, 3:02 pm
From: 松本武
Date: Tue, 21 Sep 2010 15:02:24 -0400
Local: Tues, Sep 21 2010 3:02 pm
Subject: [Shib-Users] goat-time with wind, pole, dragon inter help?
Forward | Print | Individual message | Show original | Report this message | Find messages by this author
This is question, engish is faulty therefore the right excused is requested. Thank google to translate to help. SORRY!!!!!

At often, the goat-time install a error is vomit. To how many times like the wind, a pole, and the dragon? Install 2,3 repeat, spank, vomit blows

14:14:01.869 - INFO
[edu.internet2.middleware.shibboleth.common.config.profile.JSPErrorHandlerBeanDefinitionParser:45]
- Parsing configuration for JSP error handler.

Not precise the vomit but with aspect similar, is vomited concealed in fold of goat-time lumber? goat-time see like the wind, pole, and dragon? This insult to father's stones? JSP error handler with wind, pole, dragon with intercourse to goat-time? Or chance lack of skill with a goat-time?

Please apologize for your stupidity. There are a many thank you

Original post here

Modified Standalone Apache Commons Codec Base64 for Use in PeopleCode

2010-09-17T11:42:00.008-04:00

Just modified Apache Commons Codec 1.4's Base64 class to be in the default package and made mods to remove all other codec dependencies and use IOException vs. EncoderException/DecoderException, and most importantly to remove interface usage and not have to implement encode(Object) and decode(Object) (required by the interface). The encode(Object) and decode(Object) overloads were messing things up when called from PeopleCode (in PeopleSoft PeopleTools v8.49.xx), I'm told.

Here is the class to save in Base64.java and then compile in the required version of Java (1.4+, I think, but I compiled in 1.5):


/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 * 
 *      http://www.apache.org/licenses/LICENSE-2.0
 * 
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

import java.io.IOException;
import java.io.UnsupportedEncodingException;
import java.math.BigInteger;


/**
 * Provides Base64 encoding and decoding as defined by RFC 2045.
 * 
 * <p>
 * This class implements section <cite>6.8. Base64 Content-Transfer-Encoding</cite> from RFC 2045 <cite>Multipurpose
 * Internet Mail Extensions (MIME) Part One: Format of Internet Message Bodies</cite> by Freed and Borenstein.
 * </p>
 * <p>
 * The class can be parameterized in the following manner with various constructors:
 * <ul>
 * <li>URL-safe mode: Default off.</li>
 * <li>Line length: Default 76. Line length that aren't multiples of 4 will still essentially end up being multiples of
 * 4 in the encoded data.
 * <li>Line separator: Default is CRLF ("\r\n")</li>
 * </ul>
 * </p>
 * <p>
 * Since this class operates directly on byte streams, and not character streams, it is hard-coded to only encode/decode
 * character encodings which are compatible with the lower 127 ASCII chart (ISO-8859-1, Windows-1252, UTF-8, etc).
 * </p>
 * Modified version of Base64 from Apache Commons Codec 1.4, by Gary S. Weaver, for use in PeopleCode that was needed at Duke University.
 *
 * Following changes were made:
 * <ul>
 * <li>Per internal request, was made standalone class and in default package.</li>
 * <li>Integrated parts of org.apache.commons.codec.StringUtils and org.apache.commons.codec.CharEncoding so it could be a single class.</li>
 * <li>Removed implementation of org.apache.commons.codec.BinaryDecoder/org.apache.commons.codec.BinaryEncoder interfaces and DecoderException/EncoderException classes,
 *     changing to use more general IOException to keep required code to a single class.</li>
 * <li>Removed decode(Object) and encode(Object) methods that were causing the following error in PeopleSoft:
 *     Calling Java %1.%2: more than one overload matches. (...)
 *     While calling the noted Java method, the PeopleCode interface found more than once method signature 
 *     that could match the given parameter types.</li>
 * <li>Removed javadoc info about revision, etc. so people wouldn't get confused and contact the wrong person about issues.</li>
 * </ul>
 */
public class Base64 {
    private static final int DEFAULT_BUFFER_RESIZE_FACTOR = 2;

    private static final int DEFAULT_BUFFER_SIZE = 8192;

    /**
     * Chunk size per RFC 2045 section 6.8.
     * 
     * <p>
     * The {@value} character limit does not count the trailing CRLF, but counts all other characters, including any
     * equal signs.
     * </p>
     * 
     * @see <a href="http://www.ietf.org/rfc/rfc2045.txt">RFC 2045 section 6.8</a>
     */
    static final int CHUNK_SIZE = 76;

    /**
     * Chunk separator per RFC 2045 section 2.1.
     *
     * <p>
     * N.B. The next major release may break compatibility and make this field private.
     * </p>
     * 
     * @see <a href="http://www.ietf.org/rfc/rfc2045.txt">RFC 2045 section 2.1</a>
     */
    static final byte[] CHUNK_SEPARATOR = {'\r', '\n'};

    /**
     * This array is a lookup table that translates 6-bit positive integer index values into their "Base64 Alphabet"
     * equivalents as specified in Table 1 of RFC 2045.
     * 
     * Thanks to "commons" project in ws.apache.org for this code.
     * http://svn.apache.org/repos/asf/webservices/commons/trunk/modules/util/
     */
    private static final byte[] STANDARD_ENCODE_TABLE = {
            'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M',
            'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z',
            'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm',
            'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z',
            '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '+', '/'
    };

    /**
     * This is a copy of the STANDARD_ENCODE_TABLE above, but with + and /
     * changed to - and _ to make the encoded Base64 results more URL-SAFE.
     * This table is only used when the Base64's mode is set to URL-SAFE.
     */    
    private static final byte[] URL_SAFE_ENCODE_TABLE = {
            'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M',
            'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z',
            'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm',
            'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z',
            '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '-', '_'
    };

    /**
     * Byte used to pad output.
     */
    private static final byte PAD = '=';

    /**
     * This array is a lookup table that translates Unicode characters drawn from the "Base64 Alphabet" (as specified in
     * Table 1 of RFC 2045) into their 6-bit positive integer equivalents. Characters that are not in the Base64
     * alphabet but fall within the bounds of the array are translated to -1.
     * 
     * Note: '+' and '-' both decode to 62. '/' and '_' both decode to 63. This means decoder seamlessly handles both
     * URL_SAFE and STANDARD base64. (The encoder, on the other hand, needs to know ahead of time what to emit).
     * 
     * Thanks to "commons" project in ws.apache.org for this code.
     * http://svn.apache.org/repos/asf/webservices/commons/trunk/modules/util/
     */
    private static final byte[] DECODE_TABLE = {
            -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
            -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
            -1, -1, -1, -1, -1, -1, -1, -1, -1, 62, -1, 62, -1, 63, 52, 53, 54,
            55, 56, 57, 58, 59, 60, 61, -1, -1, -1, -1, -1, -1, -1, 0, 1, 2, 3, 4,
            5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23,
            24, 25, -1, -1, -1, -1, 63, -1, 26, 27, 28, 29, 30, 31, 32, 33, 34,
            35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51
    };

    /** Mask used to extract 6 bits, used when encoding */
    private static final int MASK_6BITS = 0x3f;

    /** Mask used to extract 8 bits, used in decoding base64 bytes */
    private static final int MASK_8BITS = 0xff;

    // The static final fields above are used for the original static byte[] methods on Base64.
    // The private member fields below are used with the new streaming approach, which requires
    // some state be preserved between calls of encode() and decode().

    /**
     * Encode table to use: either STANDARD or URL_SAFE. Note: the DECODE_TABLE above remains static because it is able
     * to decode both STANDARD and URL_SAFE streams, but the encodeTable must be a member variable so we can switch
     * between the two modes.
     */
    private final byte[] encodeTable;

    /**
     * Line length for encoding. Not used when decoding. A value of zero or less implies no chunking of the base64
     * encoded data.
     */
    private final int lineLength;

    /**
     * Line separator for encoding. Not used when decoding. Only used if lineLength > 0.
     */
    private final byte[] lineSeparator;

    /**
     * Convenience variable to help us determine when our buffer is going to run out of room and needs resizing.
     * <code>decodeSize = 3 + lineSeparator.length;</code>
     */
    private final int decodeSize;

    /**
     * Convenience variable to help us determine when our buffer is going to run out of room and needs resizing.
     * <code>encodeSize = 4 + lineSeparator.length;</code>
     */
    private final int encodeSize;

    /**
     * Buffer for streaming.
     */
    private byte[] buffer;

    /**
     * Position where next character should be written in the buffer.
     */
    private int pos;

    /**
     * Position where next character should be read from the buffer.
     */
    private int readPos;

    /**
     * Variable tracks how many characters have been written to the current line. Only used when encoding. We use it to
     * make sure each encoded line never goes beyond lineLength (if lineLength > 0).
     */
    private int currentLinePos;

    /**
     * Writes to the buffer only occur after every 3 reads when encoding, an every 4 reads when decoding. This variable
     * helps track that.
     */
    private int modulus;

    /**
     * Boolean flag to indicate the EOF has been reached. Once EOF has been reached, this Base64 object becomes useless,
     * and must be thrown away.
     */
    private boolean eof;

    /**
     * Place holder for the 3 bytes we're dealing with for our base64 logic. Bitwise operations store and extract the
     * base64 encoding or decoding from this variable.
     */
    private int x;

    /**
     * Creates a Base64 codec used for decoding (all modes) and encoding in URL-unsafe mode.
     * <p>
     * When encoding the line length is 76, the line separator is CRLF, and the encoding table is STANDARD_ENCODE_TABLE.
     * </p>
     * 
     * <p>
     * When decoding all variants are supported.
     * </p>
     */
    public Base64() {
        this(false);
    }

    /**
     * Creates a Base64 codec used for decoding (all modes) and encoding in the given URL-safe mode.
     * <p>
     * When encoding the line length is 76, the line separator is CRLF, and the encoding table is STANDARD_ENCODE_TABLE.
     * </p>
     * 
     * <p>
     * When decoding all variants are supported.
     * </p>
     * 
     * @param urlSafe
     *            if <code>true</code>, URL-safe encoding is used. In most cases this should be set to
     *            <code>false</code>.
     * @since 1.4
     */
    public Base64(boolean urlSafe) {
        this(CHUNK_SIZE, CHUNK_SEPARATOR, urlSafe);
    }

    /**
     * Creates a Base64 codec used for decoding (all modes) and encoding in URL-unsafe mode.
     * <p>
     * When encoding the line length is given in the constructor, the line separator is CRLF, and the encoding table is
     * STANDARD_ENCODE_TABLE.
     * </p>
     * <p>
     * Line lengths that aren't multiples of 4 will still essentially end up being multiples of 4 in the encoded data.
     * </p>
     * <p>
     * When decoding all variants are supported.
     * </p>
     * 
     * @param lineLength
     *            Each line of encoded data will be at most of the given length (rounded down to nearest multiple of 4).
     *            If lineLength <= 0, then the output will not be divided into lines (chunks). Ignored when decoding.
     * @since 1.4
     */
    public Base64(int lineLength) {
        this(lineLength, CHUNK_SEPARATOR);
    }

    /**
     * Creates a Base64 codec used for decoding (all modes) and encoding in URL-unsafe mode.
     * <p>
     * When encoding the line length and line separator are given in the constructor, and the encoding table is
     * STANDARD_ENCODE_TABLE.
     * </p>
     * <p>
     * Line lengths that aren't multiples of 4 will still essentially end up being multiples of 4 in the encoded data.
     * </p>
     * <p>
     * When decoding all variants are supported.
     * </p>
     * 
     * @param lineLength
     *            Each line of encoded data will be at most of the given length (rounded down to nearest multiple of 4).
     *            If lineLength <= 0, then the output will not be divided into lines (chunks). Ignored when decoding.
     * @param lineSeparator
     *            Each line of encoded data will end with this sequence of bytes.
     * @throws IllegalArgumentException
     *             Thrown when the provided lineSeparator included some base64 characters.
     * @since 1.4
     */
    public Base64(int lineLength, byte[] lineSeparator) {
        this(lineLength, lineSeparator, false);
    }

    /**
     * Creates a Base64 codec used for decoding (all modes) and encoding in URL-unsafe mode.
     * <p>
     * When encoding the line length and line separator are given in the constructor, and the encoding table is
     * STANDARD_ENCODE_TABLE.
     * </p>
     * <p>
     * Line lengths that aren't multiples of 4 will still essentially end up being multiples of 4 in the encoded data.
     * </p>
     * <p>
     * When decoding all variants are supported.
     * </p>
     * 
     * @param lineLength
     *            Each line of encoded data will be at most of the given length (rounded down to nearest multiple of 4).
     *            If lineLength <= 0, then the output will not be divided into lines (chunks). Ignored when decoding.
     * @param lineSeparator
     *            Each line of encoded data will end with this sequence of bytes.
     * @param urlSafe
     *            Instead of emitting '+' and '/' we emit '-' and '_' respectively. urlSafe is only applied to encode
     *            operations. Decoding seamlessly handles both modes.
     * @throws IllegalArgumentException
     *             The provided lineSeparator included some base64 characters. That's not going to work!
     * @since 1.4
     */
    public Base64(int lineLength, byte[] lineSeparator, boolean urlSafe) {
        if (lineSeparator == null) {
            lineLength = 0;  // disable chunk-separating
            lineSeparator = CHUNK_SEPARATOR;  // this just gets ignored
        }
        this.lineLength = lineLength > 0 ? (lineLength / 4) * 4 : 0;
        this.lineSeparator = new byte[lineSeparator.length];
        System.arraycopy(lineSeparator, 0, this.lineSeparator, 0, lineSeparator.length);
        if (lineLength > 0) {
            this.encodeSize = 4 + lineSeparator.length;
        } else {
            this.encodeSize = 4;
        }
        this.decodeSize = this.encodeSize - 1;
        if (containsBase64Byte(lineSeparator)) {
            String sep = Base64.newStringUtf8(lineSeparator);
            throw new IllegalArgumentException("lineSeperator must not contain base64 characters: [" + sep + "]");
        }
        this.encodeTable = urlSafe ? URL_SAFE_ENCODE_TABLE : STANDARD_ENCODE_TABLE;
    }

    /**
     * Returns our current encode mode. True if we're URL-SAFE, false otherwise.
     * 
     * @return true if we're in URL-SAFE mode, false otherwise.
     * @since 1.4
     */
    public boolean isUrlSafe() {
        return this.encodeTable == URL_SAFE_ENCODE_TABLE;
    }

    /**
     * Returns true if this Base64 object has buffered data for reading.
     * 
     * @return true if there is Base64 object still available for reading.
     */
    boolean hasData() {
        return this.buffer != null;
    }

    /**
     * Returns the amount of buffered data available for reading.
     * 
     * @return The amount of buffered data available for reading.
     */
    int avail() {
        return buffer != null ? pos - readPos : 0;
    }

    /** Doubles our buffer. */
    private void resizeBuffer() {
        if (buffer == null) {
            buffer = new byte[DEFAULT_BUFFER_SIZE];
            pos = 0;
            readPos = 0;
        } else {
            byte[] b = new byte[buffer.length * DEFAULT_BUFFER_RESIZE_FACTOR];
            System.arraycopy(buffer, 0, b, 0, buffer.length);
            buffer = b;
        }
    }

    /**
     * Extracts buffered data into the provided byte[] array, starting at position bPos, up to a maximum of bAvail
     * bytes. Returns how many bytes were actually extracted.
     * 
     * @param b
     *            byte[] array to extract the buffered data into.
     * @param bPos
     *            position in byte[] array to start extraction at.
     * @param bAvail
     *            amount of bytes we're allowed to extract. We may extract fewer (if fewer are available).
     * @return The number of bytes successfully extracted into the provided byte[] array.
     */
    int readResults(byte[] b, int bPos, int bAvail) {
        if (buffer != null) {
            int len = Math.min(avail(), bAvail);
            if (buffer != b) {
                System.arraycopy(buffer, readPos, b, bPos, len);
                readPos += len;
                if (readPos >= pos) {
                    buffer = null;
                }
            } else {
                // Re-using the original consumer's output array is only
                // allowed for one round.
                buffer = null;
            }
            return len;
        }
        return eof ? -1 : 0;
    }

    /**
     * Sets the streaming buffer. This is a small optimization where we try to buffer directly to the consumer's output
     * array for one round (if the consumer calls this method first) instead of starting our own buffer.
     * 
     * @param out
     *            byte[] array to buffer directly to.
     * @param outPos
     *            Position to start buffering into.
     * @param outAvail
     *            Amount of bytes available for direct buffering.
     */
    void setInitialBuffer(byte[] out, int outPos, int outAvail) {
        // We can re-use consumer's original output array under
        // special circumstances, saving on some System.arraycopy().
        if (out != null && out.length == outAvail) {
            buffer = out;
            pos = outPos;
            readPos = outPos;
        }
    }

    /**
     * <p>
     * Encodes all of the provided data, starting at inPos, for inAvail bytes. Must be called at least twice: once with
     * the data to encode, and once with inAvail set to "-1" to alert encoder that EOF has been reached, so flush last
     * remaining bytes (if not multiple of 3).
     * </p>
     * <p>
     * Thanks to "commons" project in ws.apache.org for the bitwise operations, and general approach.
     * http://svn.apache.org/repos/asf/webservices/commons/trunk/modules/util/
     * </p>
     * 
     * @param in
     *            byte[] array of binary data to base64 encode.
     * @param inPos
     *            Position to start reading data from.
     * @param inAvail
     *            Amount of bytes available from input for encoding.
     */
    void encode(byte[] in, int inPos, int inAvail) {
        if (eof) {
            return;
        }
        // inAvail < 0 is how we're informed of EOF in the underlying data we're
        // encoding.
        if (inAvail < 0) {
            eof = true;
            if (buffer == null || buffer.length - pos < encodeSize) {
                resizeBuffer();
            }
            switch (modulus) {
                case 1 :
                    buffer[pos++] = encodeTable[(x >> 2) & MASK_6BITS];
                    buffer[pos++] = encodeTable[(x << 4) & MASK_6BITS];
                    // URL-SAFE skips the padding to further reduce size.
                    if (encodeTable == STANDARD_ENCODE_TABLE) {
                        buffer[pos++] = PAD;
                        buffer[pos++] = PAD;
                    }
                    break;

                case 2 :
                    buffer[pos++] = encodeTable[(x >> 10) & MASK_6BITS];
                    buffer[pos++] = encodeTable[(x >> 4) & MASK_6BITS];
                    buffer[pos++] = encodeTable[(x << 2) & MASK_6BITS];
                    // URL-SAFE skips the padding to further reduce size.
                    if (encodeTable == STANDARD_ENCODE_TABLE) {
                        buffer[pos++] = PAD;
                    }
                    break;
            }
            if (lineLength > 0 && pos > 0) {
                System.arraycopy(lineSeparator, 0, buffer, pos, lineSeparator.length);
                pos += lineSeparator.length;
            }
        } else {
            for (int i = 0; i < inAvail; i++) {
                if (buffer == null || buffer.length - pos < encodeSize) {
                    resizeBuffer();
                }
                modulus = (++modulus) % 3;
                int b = in[inPos++];
                if (b < 0) {
                    b += 256;
                }
                x = (x << 8) + b;
                if (0 == modulus) {
                    buffer[pos++] = encodeTable[(x >> 18) & MASK_6BITS];
                    buffer[pos++] = encodeTable[(x >> 12) & MASK_6BITS];
                    buffer[pos++] = encodeTable[(x >> 6) & MASK_6BITS];
                    buffer[pos++] = encodeTable[x & MASK_6BITS];
                    currentLinePos += 4;
                    if (lineLength > 0 && lineLength <= currentLinePos) {
                        System.arraycopy(lineSeparator, 0, buffer, pos, lineSeparator.length);
                        pos += lineSeparator.length;
                        currentLinePos = 0;
                    }
                }
            }
        }
    }

    /**
     * <p>
     * Decodes all of the provided data, starting at inPos, for inAvail bytes. Should be called at least twice: once
     * with the data to decode, and once with inAvail set to "-1" to alert decoder that EOF has been reached. The "-1"
     * call is not necessary when decoding, but it doesn't hurt, either.
     * </p>
     * <p>
     * Ignores all non-base64 characters. This is how chunked (e.g. 76 character) data is handled, since CR and LF are
     * silently ignored, but has implications for other bytes, too. This method subscribes to the garbage-in,
     * garbage-out philosophy: it will not check the provided data for validity.
     * </p>
     * <p>
     * Thanks to "commons" project in ws.apache.org for the bitwise operations, and general approach.
     * http://svn.apache.org/repos/asf/webservices/commons/trunk/modules/util/
     * </p>
     * 
     * @param in
     *            byte[] array of ascii data to base64 decode.
     * @param inPos
     *            Position to start reading data from.
     * @param inAvail
     *            Amount of bytes available from input for encoding.
     */
    void decode(byte[] in, int inPos, int inAvail) {
        if (eof) {
            return;
        }
        if (inAvail < 0) {
            eof = true;
        }
        for (int i = 0; i < inAvail; i++) {
            if (buffer == null || buffer.length - pos < decodeSize) {
                resizeBuffer();
            }
            byte b = in[inPos++];
            if (b == PAD) {
                // We're done.
                eof = true;
                break;
            } else {
                if (b >= 0 && b < DECODE_TABLE.length) {
                    int result = DECODE_TABLE[b];
                    if (result >= 0) {
                        modulus = (++modulus) % 4;
                        x = (x << 6) + result;
                        if (modulus == 0) {
                            buffer[pos++] = (byte) ((x >> 16) & MASK_8BITS);
                            buffer[pos++] = (byte) ((x >> 8) & MASK_8BITS);
                            buffer[pos++] = (byte) (x & MASK_8BITS);
                        }
                    }
                }
            }
        }

        // Two forms of EOF as far as base64 decoder is concerned: actual
        // EOF (-1) and first time '=' character is encountered in stream.
        // This approach makes the '=' padding characters completely optional.
        if (eof && modulus != 0) {
            x = x << 6;
            switch (modulus) {
                case 2 :
                    x = x << 6;
                    buffer[pos++] = (byte) ((x >> 16) & MASK_8BITS);
                    break;
                case 3 :
                    buffer[pos++] = (byte) ((x >> 16) & MASK_8BITS);
                    buffer[pos++] = (byte) ((x >> 8) & MASK_8BITS);
                    break;
            }
        }
    }

    /**
     * Returns whether or not the <code>octet</code> is in the base 64 alphabet.
     * 
     * @param octet
     *            The value to test
     * @return <code>true</code> if the value is defined in the the base 64 alphabet, <code>false</code> otherwise.
     * @since 1.4
     */
    public static boolean isBase64(byte octet) {
        return octet == PAD || (octet >= 0 && octet < DECODE_TABLE.length && DECODE_TABLE[octet] != -1);
    }

    /**
     * Tests a given byte array to see if it contains only valid characters within the Base64 alphabet. Currently the
     * method treats whitespace as valid.
     * 
     * @param arrayOctet
     *            byte array to test
     * @return <code>true</code> if all bytes are valid characters in the Base64 alphabet or if the byte array is empty;
     *         false, otherwise
     */
    public static boolean isArrayByteBase64(byte[] arrayOctet) {
        for (int i = 0; i < arrayOctet.length; i++) {
            if (!isBase64(arrayOctet[i]) && !isWhiteSpace(arrayOctet[i])) {
                return false;
            }
        }
        return true;
    }

    /**
     * Tests a given byte array to see if it contains only valid characters within the Base64 alphabet.
     * 
     * @param arrayOctet
     *            byte array to test
     * @return <code>true</code> if any byte is a valid character in the Base64 alphabet; false herwise
     */
    private static boolean containsBase64Byte(byte[] arrayOctet) {
        for (int i = 0; i < arrayOctet.length; i++) {
            if (isBase64(arrayOctet[i])) {
                return true;
            }
        }
        return false;
    }

    /**
     * Encodes binary data using the base64 algorithm but does not chunk the output.
     * 
     * @param binaryData
     *            binary data to encode
     * @return byte[] containing Base64 characters in their UTF-8 representation.
     */
    public static byte[] encodeBase64(byte[] binaryData) {
        return encodeBase64(binaryData, false);
    }

    /**
     * Encodes binary data using the base64 algorithm into 76 character blocks separated by CRLF.
     *
     * @param binaryData
     *            binary data to encode
     * @return String containing Base64 characters.
     * @since 1.4
     */    
    public static String encodeBase64String(byte[] binaryData) {
        return Base64.newStringUtf8(encodeBase64(binaryData, true));
    }
    
    /**
     * Encodes binary data using a URL-safe variation of the base64 algorithm but does not chunk the output. The
     * url-safe variation emits - and _ instead of + and / characters.
     * 
     * @param binaryData
     *            binary data to encode
     * @return byte[] containing Base64 characters in their UTF-8 representation.
     * @since 1.4
     */
    public static byte[] encodeBase64URLSafe(byte[] binaryData) {
        return encodeBase64(binaryData, false, true);
    }

    /**
     * Encodes binary data using a URL-safe variation of the base64 algorithm but does not chunk the output. The
     * url-safe variation emits - and _ instead of + and / characters.
     *
     * @param binaryData
     *            binary data to encode
     * @return String containing Base64 characters
     * @since 1.4
     */    
    public static String encodeBase64URLSafeString(byte[] binaryData) {
        return Base64.newStringUtf8(encodeBase64(binaryData, false, true));
    }    

    /**
     * Encodes binary data using the base64 algorithm and chunks the encoded output into 76 character blocks
     * 
     * @param binaryData
     *            binary data to encode
     * @return Base64 characters chunked in 76 character blocks
     */
    public static byte[] encodeBase64Chunked(byte[] binaryData) {
        return encodeBase64(binaryData, true);
    }

    /**
     * Decodes a String containing containing characters in the Base64 alphabet.
     *
     * @param pArray
     *            A String containing Base64 character data
     * @return a byte array containing binary data
     * @since 1.4
     */
    public byte[] decode(String pArray) {
        return decode(Base64.getBytesUtf8(pArray));
    }

    /**
     * Decodes a byte[] containing containing characters in the Base64 alphabet.
     * 
     * @param pArray
     *            A byte array containing Base64 character data
     * @return a byte array containing binary data
     */
    public byte[] decode(byte[] pArray) {
        reset();
        if (pArray == null || pArray.length == 0) {
            return pArray;
        }
        long len = (pArray.length * 3) / 4;
        byte[] buf = new byte[(int) len];
        setInitialBuffer(buf, 0, buf.length);
        decode(pArray, 0, pArray.length);
        decode(pArray, 0, -1); // Notify decoder of EOF.

        // Would be nice to just return buf (like we sometimes do in the encode
        // logic), but we have no idea what the line-length was (could even be
        // variable).  So we cannot determine ahead of time exactly how big an
        // array is necessary.  Hence the need to construct a 2nd byte array to
        // hold the final result:

        byte[] result = new byte[pos];
        readResults(result, 0, result.length);
        return result;
    }

    /**
     * Encodes binary data using the base64 algorithm, optionally chunking the output into 76 character blocks.
     * 
     * @param binaryData
     *            Array containing binary data to encode.
     * @param isChunked
     *            if <code>true</code> this encoder will chunk the base64 output into 76 character blocks
     * @return Base64-encoded data.
     * @throws IllegalArgumentException
     *             Thrown when the input array needs an output array bigger than {@link Integer#MAX_VALUE}
     */
    public static byte[] encodeBase64(byte[] binaryData, boolean isChunked) {
        return encodeBase64(binaryData, isChunked, false);
    }

    /**
     * Encodes binary data using the base64 algorithm, optionally chunking the output into 76 character blocks.
     * 
     * @param binaryData
     *            Array containing binary data to encode.
     * @param isChunked
     *            if <code>true</code> this encoder will chunk the base64 output into 76 character blocks
     * @param urlSafe
     *            if <code>true</code> this encoder will emit - and _ instead of the usual + and / characters.
     * @return Base64-encoded data.
     * @throws IllegalArgumentException
     *             Thrown when the input array needs an output array bigger than {@link Integer#MAX_VALUE}
     * @since 1.4
     */
    public static byte[] encodeBase64(byte[] binaryData, boolean isChunked, boolean urlSafe) {
        return encodeBase64(binaryData, isChunked, urlSafe, Integer.MAX_VALUE);
    }

    /**
     * Encodes binary data using the base64 algorithm, optionally chunking the output into 76 character blocks.
     * 
     * @param binaryData
     *            Array containing binary data to encode.
     * @param isChunked
     *            if <code>true</code> this encoder will chunk the base64 output into 76 character blocks
     * @param urlSafe
     *            if <code>true</code> this encoder will emit - and _ instead of the usual + and / characters.
     * @param maxResultSize
     *            The maximum result size to accept.
     * @return Base64-encoded data.
     * @throws IllegalArgumentException
     *             Thrown when the input array needs an output array bigger than maxResultSize
     * @since 1.4
     */
    public static byte[] encodeBase64(byte[] binaryData, boolean isChunked, boolean urlSafe, int maxResultSize) {
        if (binaryData == null || binaryData.length == 0) {
            return binaryData;
        }

        long len = getEncodeLength(binaryData, CHUNK_SIZE, CHUNK_SEPARATOR);
        if (len > maxResultSize) {
            throw new IllegalArgumentException("Input array too big, the output array would be bigger (" +
                len +
                ") than the specified maxium size of " +
                maxResultSize);
        }
                
        Base64 b64 = isChunked ? new Base64(urlSafe) : new Base64(0, CHUNK_SEPARATOR, urlSafe);
        return b64.encode(binaryData);
    }

    /**
     * Decodes a Base64 String into octets
     *
     * @param base64String
     *            String containing Base64 data
     * @return Array containing decoded data.
     * @since 1.4
     */
    public static byte[] decodeBase64(String base64String) {
        return new Base64().decode(base64String);
    }

    /**
     * Decodes Base64 data into octets
     * 
     * @param base64Data
     *            Byte array containing Base64 data
     * @return Array containing decoded data.
     */
    public static byte[] decodeBase64(byte[] base64Data) {
        return new Base64().decode(base64Data);
    }

    /**
     * Discards any whitespace from a base-64 encoded block.
     * 
     * @param data
     *            The base-64 encoded data to discard the whitespace from.
     * @return The data, less whitespace (see RFC 2045).
     * @deprecated This method is no longer needed
     */
    static byte[] discardWhitespace(byte[] data) {
        byte groomedData[] = new byte[data.length];
        int bytesCopied = 0;
        for (int i = 0; i < data.length; i++) {
            switch (data[i]) {
                case ' ' :
                case '\n' :
                case '\r' :
                case '\t' :
                    break;
                default :
                    groomedData[bytesCopied++] = data[i];
            }
        }
        byte packedData[] = new byte[bytesCopied];
        System.arraycopy(groomedData, 0, packedData, 0, bytesCopied);
        return packedData;
    }

    /**
     * Checks if a byte value is whitespace or not.
     * 
     * @param byteToCheck
     *            the byte to check
     * @return true if byte is whitespace, false otherwise
     */
    private static boolean isWhiteSpace(byte byteToCheck) {
        switch (byteToCheck) {
            case ' ' :
            case '\n' :
            case '\r' :
            case '\t' :
                return true;
            default :
                return false;
        }
    }

    /**
     * Encodes a byte[] containing binary data, into a String containing characters in the Base64 alphabet.
     *
     * @param pArray
     *            a byte array containing binary data
     * @return A String containing only Base64 character data
     * @since 1.4
     */    
    public String encodeToString(byte[] pArray) {
        return Base64.newStringUtf8(encode(pArray));
    }

    /**
     * Encodes a byte[] containing binary data, into a byte[] containing characters in the Base64 alphabet.
     * 
     * @param pArray
     *            a byte array containing binary data
     * @return A byte array containing only Base64 character data
     */
    public byte[] encode(byte[] pArray) {
        reset();        
        if (pArray == null || pArray.length == 0) {
            return pArray;
        }
        long len = getEncodeLength(pArray, lineLength, lineSeparator);
        byte[] buf = new byte[(int) len];
        setInitialBuffer(buf, 0, buf.length);
        encode(pArray, 0, pArray.length);
        encode(pArray, 0, -1); // Notify encoder of EOF.
        // Encoder might have resized, even though it was unnecessary.
        if (buffer != buf) {
            readResults(buf, 0, buf.length);
        }
        // In URL-SAFE mode we skip the padding characters, so sometimes our
        // final length is a bit smaller.
        if (isUrlSafe() && pos < buf.length) {
            byte[] smallerBuf = new byte[pos];
            System.arraycopy(buf, 0, smallerBuf, 0, pos);
            buf = smallerBuf;
        }
        return buf;        
    }

    /**
     * Pre-calculates the amount of space needed to base64-encode the supplied array.
     *
     * @param pArray byte[] array which will later be encoded
     * @param chunkSize line-length of the output (<= 0 means no chunking) between each
     *        chunkSeparator (e.g. CRLF).
     * @param chunkSeparator the sequence of bytes used to separate chunks of output (e.g. CRLF).
     *
     * @return amount of space needed to encoded the supplied array.  Returns
     *         a long since a max-len array will require Integer.MAX_VALUE + 33%.
     */
    private static long getEncodeLength(byte[] pArray, int chunkSize, byte[] chunkSeparator) {
        // base64 always encodes to multiples of 4.
        chunkSize = (chunkSize / 4) * 4;

        long len = (pArray.length * 4) / 3;
        long mod = len % 4;
        if (mod != 0) {
            len += 4 - mod;
        }
        if (chunkSize > 0) {
            boolean lenChunksPerfectly = len % chunkSize == 0;
            len += (len / chunkSize) * chunkSeparator.length;
            if (!lenChunksPerfectly) {
                len += chunkSeparator.length;
            }
        }
        return len;
    }

    // Implementation of integer encoding used for crypto
    /**
     * Decodes a byte64-encoded integer according to crypto standards such as W3C's XML-Signature
     * 
     * @param pArray
     *            a byte array containing base64 character data
     * @return A BigInteger
     * @since 1.4
     */
    public static BigInteger decodeInteger(byte[] pArray) {
        return new BigInteger(1, decodeBase64(pArray));
    }

    /**
     * Encodes to a byte64-encoded integer according to crypto standards such as W3C's XML-Signature
     * 
     * @param bigInt
     *            a BigInteger
     * @return A byte array containing base64 character data
     * @throws NullPointerException
     *             if null is passed in
     * @since 1.4
     */
    public static byte[] encodeInteger(BigInteger bigInt) {
        if (bigInt == null) {
            throw new NullPointerException("encodeInteger called with null parameter");
        }
        return encodeBase64(toIntegerBytes(bigInt), false);
    }

    /**
     * Returns a byte-array representation of a <code>BigInteger</code> without sign bit.
     * 
     * @param bigInt
     *            <code>BigInteger</code> to be converted
     * @return a byte array representation of the BigInteger parameter
     */
    static byte[] toIntegerBytes(BigInteger bigInt) {
        int bitlen = bigInt.bitLength();
        // round bitlen
        bitlen = ((bitlen + 7) >> 3) << 3;
        byte[] bigBytes = bigInt.toByteArray();

        if (((bigInt.bitLength() % 8) != 0) && (((bigInt.bitLength() / 8) + 1) == (bitlen / 8))) {
            return bigBytes;
        }
        // set up params for copying everything but sign bit
        int startSrc = 0;
        int len = bigBytes.length;

        // if bigInt is exactly byte-aligned, just skip signbit in copy
        if ((bigInt.bitLength() % 8) == 0) {
            startSrc = 1;
            len--;
        }
        int startDst = bitlen / 8 - len; // to pad w/ nulls as per spec
        byte[] resizedBytes = new byte[bitlen / 8];
        System.arraycopy(bigBytes, startSrc, resizedBytes, startDst, len);
        return resizedBytes;
    }

    /**
     * Resets this Base64 object to its initial newly constructed state.
     */
    private void reset() {
        buffer = null;
        pos = 0;
        readPos = 0;
        currentLinePos = 0;
        modulus = 0;
        eof = false;
    }

    
    // The following was moved here from org.apache.commons.codec.CharEncoding. Also modified to reference this class.
    
    /**
     * <p>
     * Eight-bit Unicode Transformation Format.
     * </p>
     * <p>
     * Every implementation of the Java platform is required to support this character encoding.
     * </p>
     * 
     * @see <a href="http://java.sun.com/j2se/1.4.2/docs/api/java/nio/charset/Charset.html">Standard charsets</a>
     */
    public static final String UTF_8 = "UTF-8";

    
    // The following was moved here from org.apache.commons.codec.StringUtils. Also modified to reference this class.

    /**
     * Constructs a new <code>String</code> by decoding the specified array of bytes using the UTF-8 charset.
     * 
     * @param bytes
     *            The bytes to be decoded into characters
     * @return A new <code>String</code> decoded from the specified array of bytes using the given charset.
     * @throws IllegalStateException
     *             Thrown when a {@link UnsupportedEncodingException} is caught, which should never happen since the
     *             charset is required.
     */
    public static String newStringUtf8(byte[] bytes) {
        return Base64.newString(bytes, Base64.UTF_8);
    }

    private static IllegalStateException newIllegalStateException(String charsetName, UnsupportedEncodingException e) {
        return new IllegalStateException(charsetName + ": " + e);
    }

    /**
     * Constructs a new <code>String</code> by decoding the specified array of bytes using the given charset.
     * <p>
     * This method catches {@link UnsupportedEncodingException} and re-throws it as {@link IllegalStateException}, which
     * should never happen for a required charset name. Use this method when the encoding is required to be in the JRE.
     * </p>
     * 
     * @param bytes
     *            The bytes to be decoded into characters
     * @param charsetName
     *            The name of a required {@link java.nio.charset.Charset}
     * @return A new <code>String</code> decoded from the specified array of bytes using the given charset.
     * @throws IllegalStateException
     *             Thrown when a {@link UnsupportedEncodingException} is caught, which should never happen for a
     *             required charset name.
     * @see CharEncoding
     * @see String#String(byte[], String)
     */
    public static String newString(byte[] bytes, String charsetName) {
        if (bytes == null) {
            return null;
        }
        try {
            return new String(bytes, charsetName);
        } catch (UnsupportedEncodingException e) {
            throw Base64.newIllegalStateException(charsetName, e);
        }
    }

    /**
     * Encodes the given string into a sequence of bytes using the UTF-8 charset, storing the result into a new byte
     * array.
     * 
     * @param string
     *            the String to encode
     * @return encoded bytes
     * @throws IllegalStateException
     *             Thrown when the charset is missing, which should be never according the the Java specification.
     * @see <a href="http://java.sun.com/j2se/1.4.2/docs/api/java/nio/charset/Charset.html">Standard charsets</a>
     * @see #getBytesUnchecked(String, String)
     */
    public static byte[] getBytesUtf8(String string) {
        return Base64.getBytesUnchecked(string, Base64.UTF_8);
    }

    /**
     * Encodes the given string into a sequence of bytes using the named charset, storing the result into a new byte
     * array.
     * <p>
     * This method catches {@link UnsupportedEncodingException} and rethrows it as {@link IllegalStateException}, which
     * should never happen for a required charset name. Use this method when the encoding is required to be in the JRE.
     * </p>
     * 
     * @param string
     *            the String to encode
     * @param charsetName
     *            The name of a required {@link java.nio.charset.Charset}
     * @return encoded bytes
     * @throws IllegalStateException
     *             Thrown when a {@link UnsupportedEncodingException} is caught, which should never happen for a
     *             required charset name.
     * @see CharEncoding
     * @see String#getBytes(String)
     */
    public static byte[] getBytesUnchecked(String string, String charsetName) {
        if (string == null) {
            return null;
        }
        try {
            return string.getBytes(charsetName);
        } catch (UnsupportedEncodingException e) {
            throw Base64.newIllegalStateException(charsetName, e);
        }
    }
}

After compiling that Java file into Base64.class and putting Base64.class into the classes dir, here is an example for how it could be called in PeopleCode that I was provided:


Local object &oDecoder = CreateJavaObject("Base64");
Local object &dOut = &oDecoder.decode(&strAttachContent);
Local File &file;
&file = GetFile(&strDocFileName, "W", "UTF8");
Local string &strPutFileName = &file.Name;
&file.Close();
Local JavaObject &FOS = CreateJavaObject("java.io.FileOutputStream", &strPutFileName);
Local JavaObject &DOS = CreateJavaObject("java.io.DataOutputStream", &FOS);
&DOS.write(&dOut, 0, &dOut.length);
&DOS.close();

Hope that helps someone else.

Summary of SQL Antipatterns Presentation

2010-09-13T16:40:00.005-04:00

The following is a summary of the presentation, "SQL Antipatterns - Tables and Queries That Don't Work", by Bill Karwin. Please see Bill's presentation for details about each of these:

http://www.scribd.com/doc/2670985/SQL-Antipatterns

Things not to do:

Comma-Separated Lists: Storing list of relations as a comma-delimited list of ids.
Multi-Column Attributes: Storing ids to related tables as additional columns in same table (e.g. product_id1, product_id2, product_id3)
Entity-Attribute-Value: Storing generic key/value data in a table (e.g. attr_name, attr_value) in cases where attributes do not have to be truly dynamic, constraints on data are not enforced by application, non-relational solutions have not been explored, etc.
Metadata Tribbles: Putting year (data) into database table name or other similar splits for performance that ignore ability to partition horizontally, vertically, or use dependent tables.
ID Required: Creating a new ID field (calling it "ID") and ignoring other ID fields that could be used as ID or composite ID or that could have more descriptive names.
Phantom Files: Storing links in database to external files that should be managed transactionally, backed up, and restored with the database.
FLOAT: Storing decimal numbers (especially money or time estimates) as inexact FLOAT data type vs. NUMERIC data type.
ENUM: Using ENUM datatype when the values may change.
Readable Passwords: Storing passwords in plain text (e.g. for purpose of providing to user upon request for password) rather than MD5 digest w/salt in application prior to storage in DB.
Ambiguous GROUP BY: Using GROUP BY and select some attributes in result without using functionally dependent attributes, GROUP_CONCAT with attribute, or using OUTER JOIN instead.
HAVING: Using HAVING clause vs. WHERE clause, not realizing order of execution for HAVING is after WHERE, SELECT, aliases, and GROUP BY (and only before ORDER BY).
Poor Man's Search Engine: Using LIKE with wildcards on both sides or RLIKE (regexp) (Note: index is used if wildcard only at end of LIKE, but not used if wildcard at beginning) rather than MySQL FULLTEXT index (MyISAM only), Lucene, or Sphinx Search (for example).
Implicit Columns: Selecting wildcard columns.
User-supplied SQL: Letting users supply SQL.
SQL Injection: Putting user-supplied data into SQL expression directly rather than filtering, parameterizing.
Parameter Facade: Reusing parameter entered by user for list used by application, like WHERE bug_id in LIST ( $user_supplied_parameter ) because you want to have application also supply as "1234,3456,5678". (basically same as SQL injection)
Pseudokey Neat Freak: Changing ID values to eliminate gaps between sequential IDs OR recycling ID values.
Session Coupling: Using persistent connections and reusing uncommitted transactions, changing/using connections character set, using DB session variables, using LAST_INSERT_ID.
Phantom Side Effects: Executing external effects (that don't obey ROLLBACK) in database triggers, stored procs, and functions, and kicking off application functionality (like emailing link with ID, etc.) without waiting on transactions to complete.

Active Record and ORA-00932: inconsistent datatypes: expected - got CLOB

2010-09-13T11:12:00.005-04:00

CLOBs can be problematic in Rails, and yet they are the default datatype for ActiveRecord text type (used for textareas) when used with Oracle. With the various named scopes, etc. you can use with ActiveRecord, even seemingly simple code like:


(@user.your_models && @user.your_models.length > 0)

with this code in YourModel:


  default_scope :conditions => {:another_model_id => AnotherModel.something.id, :deleted_at => nil}
  belongs_to :another_model

  has_many :some_join_models  
  has_many :users, :through => :some_join_models, :uniq => true

can cause errors like:


OCIError: ORA-00932: inconsistent datatypes: expected - got CLOB: SELECT DISTINCT your_models.* FROM your_models  INNER JOIN some_join_models ON your_models.id = some_join_models.your_model_id    WHERE ((some_join_models.user_id = 1234)) AND (your_models.deleted_at IS NULL AND your_models.another_model_id = 6789) 

RAILS_ROOT: ...
Application Trace | Framework Trace | Full Trace

.../vendor/rails/activerecord/lib/active_record/connection_adapters/abstract_adapter.rb:212:in `log'
.../vendor/gems/activerecord-oracle_enhanced-adapter-1.2.0/lib/active_record/connection_adapters/oracle_enhanced_adapter.rb:927:in `select'
.../vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:7:in `select_all_without_query_cache'
.../vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/query_cache.rb:60:in `select_all'
.../vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/query_cache.rb:81:in `cache_sql'
.../vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/query_cache.rb:60:in `select_all'
.../vendor/rails/activerecord/lib/active_record/base.rb:661:in `find_by_sql'
.../vendor/rails/activerecord/lib/active_record/base.rb:1553:in `find_every'
.../vendor/rails/activerecord/lib/active_record/base.rb:615:in `find'
.../vendor/rails/activerecord/lib/active_record/associations/has_many_through_association.rb:73:in `find_target'
.../vendor/rails/activerecord/lib/active_record/associations/association_collection.rb:349:in `load_target'
.../vendor/rails/activerecord/lib/active_record/associations/association_collection.rb:283:in `length'
.../app/controllers/application_controller.rb:34:in `a_method_that_just_reference_all_models'
.../app/controllers/application_controller.rb:12:in `a_method'

So as a first thought, why not convert the CLOB to VARCHAR2? It's faster to access. But, since many CRUD UI and form helper tools assume that you are using text for textarea and string for text input, if you change a text type to a string type, you might have more complexity in displaying the textarea as you need it to for that field. If you want to convert to VARCHAR2 anyway, see this snippet. However, you might instead look into how you can adjust things to specify references to the CLOB column as "DBMS_LOB.SUBSTR(name_of_column)". We used that previously to get sort_by in our CRUD tool working.

The problem was the use of uniq => true in all related models. It was there to try to keep additional associations from being displayed that were duplicate, but it wasn't even doing that correctly (probably because of the unique id on the association model, SomeJoinModel, making all association records unique).

Ruby Script to Identify CLOBs and Required Length of VARCHAR2 for Conversion

2010-09-10T13:54:00.015-04:00

All text type fields in Rails are stored in Oracle as CLOBs (at time of writing), so finding CLOB types may be as easy as looking in db/schema.rb and looking for text type fields. But what if you want to find the length of the largest value of those CLOBs, or are thinking about converting them to string types?

The following script can be saved to loads/save_me_from_the_clob.rb and then loaded via script/console and load 'loads/save_me_from_the_clob.rb'. It will list all DB tables with CLOBs with the required length of VARCHAR2 based on the current maximum value/length of the CLOB column, so you wouldn't truncate any existing values if you converted them, in theory.


# avoid: ActiveRecord::StatementInvalid (OCIError: ORA-00932: inconsistent datatypes: expected - got CLOB
# by indentifying CLOB columns and their maximum values, which can be used to create a migration.

# Written by Gary S. Weaver 9/10/2010

class MaxLengthUtility < ActiveRecord::Base

  # remove the database table dependency
  def self.columns() @columns ||= []; end
  
  def self.find_max_length(table_name, column_name)
    sql = ''
    begin
      # Note: commented out version will cause errors with length > 3999, I think
      #sql = <<-SQL_END
      #SELECT * FROM (SELECT LENGTH(DBMS_LOB.SUBSTR(#{column_name})) AS "LENGTH" FROM #{table_name} WHERE LENGTH(DBMS_LOB.SUBSTR(#{column_name})) IS NOT NULL ORDER BY LENGTH(DBMS_LOB.SUBSTR(#{column_name})) DESC) WHERE ROWNUM = 1
      #SQL_END
      sql = <<-SQL_END
      SELECT * FROM (
        SELECT DBMS_LOB.GETLENGTH(#{column_name}) AS "LENGTH" 
        FROM #{table_name} 
        WHERE DBMS_LOB.GETLENGTH(#{column_name}) IS NOT NULL 
        ORDER BY DBMS_LOB.GETLENGTH(#{column_name}) DESC) 
      WHERE ROWNUM = 1
      SQL_END
      resultset = self.find_by_sql(sql)
      resultset.empty? ? nil : resultset[0].try(:length).try(:to_i)
    rescue => ex
      puts "Failed for '#{sql}'\nwith #{ex.class}: #{ex.backtrace}"
    end
  end
end

ActiveRecord::Base.connection.tables.each do |table_name|
  begin
    #puts "classifying #{table_name}"
    model = nil
    model_name = table_name.classify
    begin
      model = eval(model_name)
    rescue => ex2
      puts "There was a table #{table_name} but no corresponding model named #{model_name}, so we're cowardly abandoning attempts to check for CLOBs in it."
    end
    if model
      #puts "iterating #{table_name} columns"
      model.columns.each do |column|
        #puts "checking #{column.name}"
        if column.sql_type == "CLOB"
          puts "#{table_name}.#{column.name} is a CLOB. Maybe it could be a VARCHAR2(#{MaxLengthUtility.find_max_length(table_name, column.name)})"
        end
      end
    end
  rescue => ex
    puts "Failed for #{table_name} with #{ex.class}: #{ex.backtrace}"
  end
end

How to Find the Maximum Length Value in a CLOB Column in Oracle

2010-09-10T11:07:00.026-04:00

If you want to convert CLOB to VARCHAR2 for speed or to avoid LOB related complication in queries, you might need to know how long the VARCHAR2 will need to be.

To determine the length of the CLOB column in characters based on getting the actual value of the CLOB (which you'd need to do if you converted to VARCHAR2), use this query:


SELECT * FROM (
  SELECT LENGTH(DBMS_LOB.SUBSTR(your_column_name)) 
  AS "LENGTH" 
  FROM your_table_name 
  WHERE LENGTH(DBMS_LOB.SUBSTR(your_column_name)) IS NOT NULL 
  ORDER BY LENGTH(DBMS_LOB.SUBSTR(your_column_name)) DESC)
WHERE ROWNUM = 1

If you're using an older version of Oracle pre-8i, you'll need to use another method than rownum = 1 to get the value with the largest size, in which case you should read:

How does one select the TOP N rows from a table?

If you get:


 [Error Code: 6502, SQL State: 65000]
 ORA-06502: PL/SQL: numeric or value error: 
 character string buffer too small
 ORA-06512: at line 1

I think it means the CLOB is larger than can be held than in the variable Oracle uses to hold the result from DBMS_LOB.SUBSTR(your_column_name). I think that max length is 4000, by default. I skimmed results against our data and length got close to but never over 3999 without an error. I believe you might be able to add an additional argument specifying length.

But if you need CLOB length regardless of whether you could convert, use DBMS_LOB.GETLENGTH(your_column_name) to get the length. I saw some say the following might work to get large substr out: DBMS_LOB.SUBSTR(your_column_name, 4000, 1), DBMS_LOB.SUBSTR(your_column_name, 4000, 4001), DBMS_LOB.SUBSTR(your_column_name, 4000, 8001), ...


SELECT * FROM (
  SELECT DBMS_LOB.GETLENGTH(my_column_name) AS "LENGTH" 
  FROM my_table_name 
  WHERE DBMS_LOB.GETLENGTH(my_column_name) IS NOT NULL 
  ORDER BY DBMS_LOB.GETLENGTH(my_column_name) DESC) 
WHERE ROWNUM = 1

See:

Oracle DBMS_LOB

How to Fix ActiveRecord::StatementInvalid (OCIError: ORA-02289: sequence does not exist: select your_model_name_seq.nextval id from dual)

2010-08-30T16:22:00.004-04:00

Was getting the following error until I noticed that I'd accidentally defined the model's table in the migration with :id => false (which accidentally crept in there, so was good that it had an error). To fix, I did rake db:rollback, removed the :id => false from the create table, and did rake rb:migrate.


ActiveRecord::StatementInvalid (OCIError: ORA-02289: sequence does not exist: select your_model_name_seq.nextval id from dual):
  vendor/gems/activerecord-oracle_enhanced-adapter-1.2.0/lib/active_record/connection_adapters/oracle_enhanced_adapter.rb:927:in `select'
  vendor/gems/activerecord-oracle_enhanced-adapter-1.2.0/lib/active_record/connection_adapters/oracle_enhanced_adapter.rb:464:in `next_sequence_value'
  /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/webrick/httpserver.rb:104:in `service'
  /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/webrick/httpserver.rb:65:in `run'
  /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'
  /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/webrick/server.rb:162:in `start'
  /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'
  /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/webrick/server.rb:95:in `start'
  /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/webrick/server.rb:92:in `each'
  /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/webrick/server.rb:92:in `start'
  /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/webrick/server.rb:23:in `start'
  /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/webrick/server.rb:82:in `start'

Make Ruby Array to_s Make Comma-delimited Strings

2010-08-26T10:44:00.001-04:00

If you want to_s when called to array to always output comma-delimited array values, try the following from a post by Ben Giddings (long ago in 2003):


$, = ", "

There's also another solution in there along with Yukihiro Matsumoto's response to both. It's somewhat annoying to always type .join(', ') when printing out an array, so I might start to use this, at least when using irb.

Straight SQL to Oracle via Ruby Script with RBI

2010-08-26T10:26:00.006-04:00

If you have some straight SQL to run against Oracle, and want to execute it in Ruby in a script sans Rails, try DBI. It's one of the older projects, but works fine for purposes of a script.

Put this into a file called sql.rb. This is heavily based on the MySQL example in Using the Ruby DBI Module by Paul DuBois rev. 1.03 (2006-11-28):


#!/usr/bin/ruby -w
# requires you to have previously run:
# gem install dbi
require 'rubygems'
require 'dbi'
puts "Loaded DBI #{DBI::VERSION}"

begin
  #format for Oracle is DBI.connect('DBI:OCI8://host:port/service_name','username','password')
  dbh = DBI.connect('DBI:OCI8://db.acme.org:1234/ACMETEST.WORLD','jdoe','passwd')

  # get server version string and display it
  row = dbh.select_one("select version from v$instance")
  puts "Server version: #{row[0]}"
rescue DBI::DatabaseError => e
  puts "An error occurred"
  puts "Error code: #{e.err}"
  puts "Error message: #{e.errstr}"
ensure
  # disconnect from server
  dbh.disconnect if dbh
end

Then:


chmod +x sql.rb

Modify that file to change the connect string. Notice that the example "ACMETEST.WORLD" is a service_name. So, if you have a service name defined in tnsnames.ora, you could use that, but you don't have to (for example: I just appended ".WORLD" to convert the SID to a service name).

If you haven't already, install dbi:


gem install dbi

Then execute:


./sql.rb

Be sure to use "prepare" to prepare statements if you can't trust the data going into the values in your query. For example:


  prepared_query = dbh.prepare("select email from user where name = ?")
  row = dbh.select_one(prepared_query, name)

For more info, see Using the Ruby DBI Module which provides some info on how to use it.

Using JMeter to Test Shibboleth Login to uPortal

2010-08-20T15:54:00.003-04:00

The following is a generified JMeter 2.4 test to use as a template so that you can test Shibboleth login to uPortal. This is based heavily on a script provided by Stan F. locally. Note that the last step in the process is to again get the uPortal Login page, as that actually logs you into uPortal, not just Shibboleth.

Save to a file like portal_test.jmx and open in JMeter. Then just edit the file and replace "portal.acme.edu" with your uPortal server, "my.shibboleth.idp.acme.edu" with your Shibboleth IdP, and the two "replace this..." strings with appropriate content. By default, it runs as a monitor (indefinitely), writing errors out as files into the bin directory of JMeter, but you can tweak it however you wish to make it a real load test, rather than a simple monitoring script.

(Note: at first I was going to use the Mailer Visualizer to send emails with info when it failed, but that action still seems fairly limited in 2.4.)


<?xml version="1.0" encoding="UTF-8"?>
<jmeterTestPlan version="1.2" properties="2.1">
  <hashTree>
    <TestPlan guiclass="TestPlanGui" testclass="TestPlan" testname="My Test Plan" enabled="true">
      <boolProp name="TestPlan.functional_mode">false</boolProp>
      <collectionProp name="TestPlan.thread_groups"/>
      <elementProp name="TestPlan.user_defined_variables" elementType="Arguments" guiclass="ArgumentsPanel" testclass="Arguments" testname="Argument List" enabled="true">
        <collectionProp name="Arguments.arguments"/>
      </elementProp>
      <boolProp name="TestPlan.serialize_threadgroups">false</boolProp>
      <stringProp name="TestPlan.comments"></stringProp>
      <stringProp name="TestPlan.user_define_classpath"></stringProp>
    </TestPlan>
    <hashTree>
      <ThreadGroup guiclass="ThreadGroupGui" testclass="ThreadGroup" testname="Thread Group" enabled="true">
        <elementProp name="ThreadGroup.main_controller" elementType="LoopController" guiclass="LoopControlPanel" testclass="LoopController" testname="Loop Controller" enabled="true">
          <boolProp name="LoopController.continue_forever">false</boolProp>
          <intProp name="LoopController.loops">-1</intProp>
        </elementProp>
        <stringProp name="ThreadGroup.num_threads">1</stringProp>
        <stringProp name="ThreadGroup.ramp_time">1</stringProp>
        <longProp name="ThreadGroup.start_time">1276016381000</longProp>
        <longProp name="ThreadGroup.end_time">1276016381000</longProp>
        <boolProp name="ThreadGroup.scheduler">false</boolProp>
        <stringProp name="ThreadGroup.on_sample_error">continue</stringProp>
        <stringProp name="ThreadGroup.duration"></stringProp>
        <stringProp name="ThreadGroup.delay"></stringProp>
      </ThreadGroup>
      <hashTree>
        <GaussianRandomTimer guiclass="GaussianRandomTimerGui" testclass="GaussianRandomTimer" testname="Gaussian Random Timer" enabled="true">
          <stringProp name="ConstantTimer.delay">300</stringProp>
          <stringProp name="RandomTimer.range">300</stringProp>
        </GaussianRandomTimer>
        <hashTree/>
        <CookieManager guiclass="CookiePanel" testclass="CookieManager" testname="HTTP Cookie Manager" enabled="true">
          <collectionProp name="CookieManager.cookies"/>
          <boolProp name="CookieManager.clearEachIteration">true</boolProp>
        </CookieManager>
        <hashTree/>
        <UserParameters guiclass="UserParametersGui" testclass="UserParameters" testname="User Parameters" enabled="true">
          <boolProp name="TestElement.per_iteration">true</boolProp>
          <collectionProp name="UserParameters.names">
            <stringProp name="VIEWSTATE">VIEWSTATE</stringProp>
            <stringProp name="jsessionid">jsessionid</stringProp>
          </collectionProp>
          <collectionProp name="UserParameters.thread_values">
            <collectionProp name="745697">
              <stringProp name=""></stringProp>
              <stringProp name=""></stringProp>
            </collectionProp>
          </collectionProp>
          <boolProp name="UserParameters.per_iteration">false</boolProp>
        </UserParameters>
        <hashTree/>
        <HeaderManager guiclass="HeaderPanel" testclass="HeaderManager" testname="HTTP Header Manager" enabled="true">
          <collectionProp name="HeaderManager.headers">
            <elementProp name="" elementType="Header">
              <stringProp name="Header.name">User-Agent</stringProp>
              <stringProp name="Header.value">Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3</stringProp>
            </elementProp>
            <elementProp name="" elementType="Header">
              <stringProp name="Header.name">Accept</stringProp>
              <stringProp name="Header.value">text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8</stringProp>
            </elementProp>
            <elementProp name="" elementType="Header">
              <stringProp name="Header.name">Accept-Language</stringProp>
              <stringProp name="Header.value">en-us,en;q=0.5</stringProp>
            </elementProp>
            <elementProp name="" elementType="Header">
              <stringProp name="Header.name">Accept-Encoding</stringProp>
              <stringProp name="Header.value">gzip,deflate</stringProp>
            </elementProp>
            <elementProp name="" elementType="Header">
              <stringProp name="Header.name">Accept-Charset</stringProp>
              <stringProp name="Header.value">ISO-8859-1,utf-8;q=0.7,*;q=0.7</stringProp>
            </elementProp>
          </collectionProp>
        </HeaderManager>
        <hashTree/>
        <LoopController guiclass="LoopControlPanel" testclass="LoopController" testname="Step 1" enabled="true">
          <boolProp name="LoopController.continue_forever">true</boolProp>
          <stringProp name="LoopController.loops">1</stringProp>
        </LoopController>
        <hashTree>
          <HTTPSampler guiclass="HttpTestSampleGui" testclass="HTTPSampler" testname="https://portal.acme.edu/uPortal" enabled="true">
            <elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" testname="Argument List" enabled="true">
              <collectionProp name="Arguments.arguments"/>
            </elementProp>
            <stringProp name="HTTPSampler.domain">portal.acme.edu</stringProp>
            <stringProp name="HTTPSampler.port">443</stringProp>
            <stringProp name="HTTPSampler.connect_timeout"></stringProp>
            <stringProp name="HTTPSampler.response_timeout"></stringProp>
            <stringProp name="HTTPSampler.protocol">https</stringProp>
            <stringProp name="HTTPSampler.contentEncoding"></stringProp>
            <stringProp name="HTTPSampler.path">/uPortal</stringProp>
            <stringProp name="HTTPSampler.method">GET</stringProp>
            <boolProp name="HTTPSampler.follow_redirects">true</boolProp>
            <boolProp name="HTTPSampler.auto_redirects">false</boolProp>
            <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
            <boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
            <stringProp name="HTTPSampler.FILE_NAME"></stringProp>
            <stringProp name="HTTPSampler.FILE_FIELD"></stringProp>
            <stringProp name="HTTPSampler.mimetype"></stringProp>
            <boolProp name="HTTPSampler.monitor">false</boolProp>
            <stringProp name="HTTPSampler.embedded_url_re"></stringProp>
          </HTTPSampler>
          <hashTree/>
          <HTTPSampler guiclass="HttpTestSampleGui" testclass="HTTPSampler" testname="https://portal.acme.edu/uPortal/Login" enabled="true">
            <elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" testname="Argument List" enabled="true">
              <collectionProp name="Arguments.arguments">
                <elementProp name="Login" elementType="HTTPArgument">
                  <stringProp name="HTTPArgument.always_encode">true</stringProp>
                  <stringProp name="Argument.value">Login</stringProp>
                  <stringProp name="Argument.name">Login</stringProp>
                  <stringProp name="Argument.metadata">=</stringProp>
                  <stringProp name="Argument.use_equals">true</stringProp>
                  <boolProp name="HTTPArgument.use_equals">true</boolProp>
                </elementProp>
              </collectionProp>
            </elementProp>
            <stringProp name="HTTPSampler.domain">portal.acme.edu</stringProp>
            <stringProp name="HTTPSampler.port">443</stringProp>
            <stringProp name="HTTPSampler.connect_timeout"></stringProp>
            <stringProp name="HTTPSampler.response_timeout"></stringProp>
            <stringProp name="HTTPSampler.protocol">https</stringProp>
            <stringProp name="HTTPSampler.contentEncoding"></stringProp>
            <stringProp name="HTTPSampler.path">/uPortal/Login</stringProp>
            <stringProp name="HTTPSampler.method">POST</stringProp>
            <boolProp name="HTTPSampler.follow_redirects">true</boolProp>
            <boolProp name="HTTPSampler.auto_redirects">false</boolProp>
            <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
            <boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
            <stringProp name="HTTPSampler.FILE_NAME"></stringProp>
            <stringProp name="HTTPSampler.FILE_FIELD"></stringProp>
            <stringProp name="HTTPSampler.mimetype"></stringProp>
            <boolProp name="HTTPSampler.monitor">false</boolProp>
            <stringProp name="HTTPSampler.embedded_url_re"></stringProp>
          </HTTPSampler>
          <hashTree/>
          <HTTPSampler guiclass="HttpTestSampleGui" testclass="HTTPSampler" testname="https://my.shibboleth.idp.acme.edu/idp/Authn/UserPassword" enabled="true">
            <elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" testname="Argument List" enabled="true">
              <collectionProp name="Arguments.arguments">
                <elementProp name="j_username" elementType="HTTPArgument">
                  <stringProp name="HTTPArgument.always_encode">true</stringProp>
                  <stringProp name="Argument.value">some_user</stringProp>
                  <stringProp name="Argument.name">j_username</stringProp>
                  <stringProp name="Argument.metadata">=</stringProp>
                  <stringProp name="Argument.use_equals">true</stringProp>
                  <boolProp name="HTTPArgument.use_equals">true</boolProp>
                </elementProp>
                <elementProp name="j_password" elementType="HTTPArgument">
                  <stringProp name="HTTPArgument.always_encode">true</stringProp>
                  <stringProp name="Argument.value">some_password</stringProp>
                  <stringProp name="Argument.name">j_password</stringProp>
                  <stringProp name="Argument.metadata">=</stringProp>
                  <stringProp name="Argument.use_equals">true</stringProp>
                  <boolProp name="HTTPArgument.use_equals">true</boolProp>
                </elementProp>
                <elementProp name="Submit" elementType="HTTPArgument">
                  <stringProp name="HTTPArgument.always_encode">true</stringProp>
                  <stringProp name="Argument.value">Enter</stringProp>
                  <stringProp name="Argument.name">Submit</stringProp>
                  <stringProp name="Argument.metadata">=</stringProp>
                  <stringProp name="Argument.use_equals">true</stringProp>
                  <boolProp name="HTTPArgument.use_equals">true</boolProp>
                </elementProp>
              </collectionProp>
            </elementProp>
            <stringProp name="HTTPSampler.domain">my.shibboleth.idp.acme.edu</stringProp>
            <stringProp name="HTTPSampler.port">443</stringProp>
            <stringProp name="HTTPSampler.connect_timeout"></stringProp>
            <stringProp name="HTTPSampler.response_timeout"></stringProp>
            <stringProp name="HTTPSampler.protocol">https</stringProp>
            <stringProp name="HTTPSampler.contentEncoding"></stringProp>
            <stringProp name="HTTPSampler.path">/idp/Authn/UserPassword</stringProp>
            <stringProp name="HTTPSampler.method">POST</stringProp>
            <boolProp name="HTTPSampler.follow_redirects">true</boolProp>
            <boolProp name="HTTPSampler.auto_redirects">false</boolProp>
            <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
            <boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
            <stringProp name="HTTPSampler.FILE_NAME"></stringProp>
            <stringProp name="HTTPSampler.FILE_FIELD"></stringProp>
            <stringProp name="HTTPSampler.mimetype"></stringProp>
            <boolProp name="HTTPSampler.monitor">false</boolProp>
            <stringProp name="HTTPSampler.embedded_url_re"></stringProp>
          </HTTPSampler>
          <hashTree>
            <XPathExtractor guiclass="XPathExtractorGui" testclass="XPathExtractor" testname="XPath Extractor" enabled="true">
              <stringProp name="XPathExtractor.default"></stringProp>
              <stringProp name="XPathExtractor.refname">samlresponse</stringProp>
              <stringProp name="XPathExtractor.xpathQuery">/html//input[@name=&quot;SAMLResponse&quot;]/@value</stringProp>
              <boolProp name="XPathExtractor.tolerant">true</boolProp>
              <boolProp name="XPathExtractor.namespace">true</boolProp>
            </XPathExtractor>
            <hashTree/>
            <XPathExtractor guiclass="XPathExtractorGui" testclass="XPathExtractor" testname="XPath Extractor" enabled="true">
              <stringProp name="XPathExtractor.default"></stringProp>
              <stringProp name="XPathExtractor.refname">relaystate</stringProp>
              <stringProp name="XPathExtractor.xpathQuery">/html/input[@name=&quot;RelayState&quot;]/@value</stringProp>
              <boolProp name="XPathExtractor.tolerant">false</boolProp>
              <boolProp name="XPathExtractor.namespace">true</boolProp>
            </XPathExtractor>
            <hashTree/>
            <ResponseAssertion guiclass="AssertionGui" testclass="ResponseAssertion" testname="Response Assertion" enabled="true">
              <collectionProp name="Asserion.test_strings">
                <stringProp name="-274265185">press the Continue button</stringProp>
              </collectionProp>
              <stringProp name="Assertion.test_field">Assertion.response_data</stringProp>
              <boolProp name="Assertion.assume_success">false</boolProp>
              <intProp name="Assertion.test_type">2</intProp>
            </ResponseAssertion>
            <hashTree/>
          </hashTree>
          <HTTPSampler guiclass="HttpTestSampleGui" testclass="HTTPSampler" testname="https://portal.acme.edu/Shibboleth.sso/SAML2/POST" enabled="true">
            <elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" testname="Argument List" enabled="true">
              <collectionProp name="Arguments.arguments">
                <elementProp name="RelayState" elementType="HTTPArgument">
                  <stringProp name="HTTPArgument.always_encode">true</stringProp>
                  <stringProp name="Argument.value">${relaystate}</stringProp>
                  <stringProp name="Argument.name">RelayState</stringProp>
                  <stringProp name="Argument.metadata">=</stringProp>
                  <stringProp name="Argument.use_equals">true</stringProp>
                  <boolProp name="HTTPArgument.use_equals">true</boolProp>
                </elementProp>
                <elementProp name="SAMLResponse" elementType="HTTPArgument">
                  <stringProp name="HTTPArgument.always_encode">true</stringProp>
                  <stringProp name="Argument.value">${samlresponse}</stringProp>
                  <stringProp name="Argument.name">SAMLResponse</stringProp>
                  <stringProp name="Argument.metadata">=</stringProp>
                  <stringProp name="Argument.use_equals">true</stringProp>
                  <boolProp name="HTTPArgument.use_equals">true</boolProp>
                </elementProp>
              </collectionProp>
            </elementProp>
            <stringProp name="HTTPSampler.domain">portal.acme.edu</stringProp>
            <stringProp name="HTTPSampler.port">443</stringProp>
            <stringProp name="HTTPSampler.connect_timeout"></stringProp>
            <stringProp name="HTTPSampler.response_timeout"></stringProp>
            <stringProp name="HTTPSampler.protocol">https</stringProp>
            <stringProp name="HTTPSampler.contentEncoding"></stringProp>
            <stringProp name="HTTPSampler.path">/Shibboleth.sso/SAML2/POST</stringProp>
            <stringProp name="HTTPSampler.method">POST</stringProp>
            <boolProp name="HTTPSampler.follow_redirects">false</boolProp>
            <boolProp name="HTTPSampler.auto_redirects">false</boolProp>
            <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
            <boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
            <stringProp name="HTTPSampler.FILE_NAME"></stringProp>
            <stringProp name="HTTPSampler.FILE_FIELD"></stringProp>
            <stringProp name="HTTPSampler.mimetype"></stringProp>
            <boolProp name="HTTPSampler.monitor">false</boolProp>
            <stringProp name="HTTPSampler.embedded_url_re"></stringProp>
          </HTTPSampler>
          <hashTree/>
          <HTTPSampler guiclass="HttpTestSampleGui" testclass="HTTPSampler" testname="https://portal.acme.edu/uPortal/Login" enabled="true">
            <elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" testname="Argument List" enabled="true">
              <collectionProp name="Arguments.arguments"/>
            </elementProp>
            <stringProp name="HTTPSampler.domain">portal.acme.edu</stringProp>
            <stringProp name="HTTPSampler.port">443</stringProp>
            <stringProp name="HTTPSampler.connect_timeout"></stringProp>
            <stringProp name="HTTPSampler.response_timeout"></stringProp>
            <stringProp name="HTTPSampler.protocol">https</stringProp>
            <stringProp name="HTTPSampler.contentEncoding"></stringProp>
            <stringProp name="HTTPSampler.path">/uPortal/Login</stringProp>
            <stringProp name="HTTPSampler.method">GET</stringProp>
            <boolProp name="HTTPSampler.follow_redirects">true</boolProp>
            <boolProp name="HTTPSampler.auto_redirects">false</boolProp>
            <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
            <boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
            <stringProp name="HTTPSampler.FILE_NAME"></stringProp>
            <stringProp name="HTTPSampler.FILE_FIELD"></stringProp>
            <stringProp name="HTTPSampler.mimetype"></stringProp>
            <boolProp name="HTTPSampler.monitor">false</boolProp>
            <stringProp name="HTTPSampler.embedded_url_re"></stringProp>
          </HTTPSampler>
          <hashTree>
            <ResponseAssertion guiclass="AssertionGui" testclass="ResponseAssertion" testname="Response Assertion" enabled="true">
              <collectionProp name="Asserion.test_strings">
                <stringProp name="33350373">replace this with text that should be in a fully loaded page</stringProp>
              </collectionProp>
              <stringProp name="Assertion.test_field">Assertion.response_data</stringProp>
              <boolProp name="Assertion.assume_success">false</boolProp>
              <intProp name="Assertion.test_type">16</intProp>
            </ResponseAssertion>
            <hashTree/>
            <ResponseAssertion guiclass="AssertionGui" testclass="ResponseAssertion" testname="Response Assertion" enabled="true">
              <collectionProp name="Asserion.test_strings">
                <stringProp name="-75219048">replace this with text from a portlet that you would only see if you logged in</stringProp>
              </collectionProp>
              <stringProp name="Assertion.test_field">Assertion.response_data</stringProp>
              <boolProp name="Assertion.assume_success">false</boolProp>
              <intProp name="Assertion.test_type">2</intProp>
            </ResponseAssertion>
            <hashTree/>
          </hashTree>
          <UniformRandomTimer guiclass="UniformRandomTimerGui" testclass="UniformRandomTimer" testname="Uniform Random Timer" enabled="true">
            <stringProp name="ConstantTimer.delay">3000</stringProp>
            <stringProp name="RandomTimer.range">15000.0</stringProp>
          </UniformRandomTimer>
          <hashTree/>
        </hashTree>
        <ResultSaver guiclass="ResultSaverGui" testclass="ResultSaver" testname="Save Responses to a file" enabled="true">
          <stringProp name="FileSaver.filename">portal_error</stringProp>
          <boolProp name="FileSaver.errorsonly">true</boolProp>
          <boolProp name="FileSaver.skipautonumber">false</boolProp>
          <boolProp name="FileSaver.skipsuffix">false</boolProp>
          <boolProp name="FileSaver.successonly">false</boolProp>
        </ResultSaver>
        <hashTree/>
      </hashTree>
      <ResultCollector guiclass="ViewResultsFullVisualizer" testclass="ResultCollector" testname="View Results Tree" enabled="false">
        <boolProp name="ResultCollector.error_logging">false</boolProp>
        <objProp>
          <name>saveConfig</name>
          <value class="SampleSaveConfiguration">
            <time>true</time>
            <latency>true</latency>
            <timestamp>true</timestamp>
            <success>true</success>
            <label>true</label>
            <code>true</code>
            <message>true</message>
            <threadName>true</threadName>
            <dataType>true</dataType>
            <encoding>false</encoding>
            <assertions>true</assertions>
            <subresults>true</subresults>
            <responseData>false</responseData>
            <samplerData>false</samplerData>
            <xml>true</xml>
            <fieldNames>false</fieldNames>
            <responseHeaders>false</responseHeaders>
            <requestHeaders>false</requestHeaders>
            <responseDataOnError>false</responseDataOnError>
            <saveAssertionResultsFailureMessage>false</saveAssertionResultsFailureMessage>
            <assertionsResultsToSave>0</assertionsResultsToSave>
            <bytes>true</bytes>
          </value>
        </objProp>
        <stringProp name="filename"></stringProp>
      </ResultCollector>
      <hashTree/>
      <ResultCollector guiclass="SummaryReport" testclass="ResultCollector" testname="Summary Report" enabled="true">
        <boolProp name="ResultCollector.error_logging">false</boolProp>
        <objProp>
          <name>saveConfig</name>
          <value class="SampleSaveConfiguration">
            <time>true</time>
            <latency>true</latency>
            <timestamp>true</timestamp>
            <success>true</success>
            <label>true</label>
            <code>true</code>
            <message>true</message>
            <threadName>true</threadName>
            <dataType>true</dataType>
            <encoding>false</encoding>
            <assertions>true</assertions>
            <subresults>true</subresults>
            <responseData>false</responseData>
            <samplerData>false</samplerData>
            <xml>true</xml>
            <fieldNames>false</fieldNames>
            <responseHeaders>false</responseHeaders>
            <requestHeaders>false</requestHeaders>
            <responseDataOnError>false</responseDataOnError>
            <saveAssertionResultsFailureMessage>false</saveAssertionResultsFailureMessage>
            <assertionsResultsToSave>0</assertionsResultsToSave>
            <bytes>true</bytes>
          </value>
        </objProp>
        <stringProp name="filename"></stringProp>
      </ResultCollector>
      <hashTree/>
    </hashTree>
  </hashTree>
</jmeterTestPlan>

Nil-Safe Sorting in Ruby Made Easy

2010-08-11T10:33:00.012-04:00

I'm not sure why nil doesn't support <=>. In most sorting cases, nil is less than everything else, so it would seem safe enough if it were extended with that. You've run into this issue if you get:


NoMethodError: You have a nil object when you didn't expect it!
You might have expected an instance of Array.
The error occurred while evaluating nil.<=>

So, what's the workaround?

Say you want to sort an array of task assignments their project name. My first hack was to write a lot of code like the following to try to get around it:


task_assignment.sort { |x,y|
  a = x.try(:project).try(:name)
  b = y.try(:project).try(:name)
  if a && b
    a <=> b
  elsif !a && !b
    0
  else
    left ? 1 : -1
  end
}

I knew this was a little long, so I spoke with Jim who showed me a much cleaner method:


task_assignment.sort_by { |ta| ta.try(:project).try(:name) || '' }

With this code, nil, a task assignment without a project, or a project without a name are equivalent to a task assignment with a project name of ''. It's not functionally equivalent to the code above it that would weigh nil even less than '', but it's fine.

Limits of Time in Ruby

2010-07-30T17:01:00.014-04:00

Was curious about the maximum Time in Ruby and minimum Time in Ruby you could specify via number to the at method, and they seem to be -67768040609636400 and 67768036191694799, at least locally for me in a 64-bit OS X 10.6 environment. The maximum number to not produce a negative year is 67767976233550799:


$ ruby -version
ruby 1.8.7 (2009-06-12 patchlevel 174) [universal-darwin10.0]
$ irb
>> Time.at(-67768040609636400)
=> Fri Jan 01 00:00:00 -0500 -2147481748
>> Time.at(-67768040609636401)
ArgumentError: localtime error
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/irb.rb:302:in `inspect'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/irb.rb:302:in `output_value'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/irb.rb:151:in `eval_input'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/irb.rb:263:in `signal_status'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/irb.rb:147:in `eval_input'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/irb/ruby-lex.rb:244:in `each_top_level_statement'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/irb/ruby-lex.rb:230:in `loop'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/irb/ruby-lex.rb:230:in `each_top_level_statement'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/irb/ruby-lex.rb:229:in `catch'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/irb/ruby-lex.rb:229:in `each_top_level_statement'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/irb.rb:146:in `eval_input'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/irb.rb:70:in `start'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/irb.rb:69:in `catch'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/irb.rb:69:in `start'
 from /usr/bin/irb:13
Maybe IRB bug!!
>> Time.at(67768036191694799)
=> Wed Dec 31 23:59:59 -0500 -2147481749
>> Time.at(67768036191694800)
ArgumentError: localtime error
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/irb.rb:302:in `inspect'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/irb.rb:302:in `output_value'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/irb.rb:151:in `eval_input'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/irb.rb:263:in `signal_status'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/irb.rb:147:in `eval_input'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/irb/ruby-lex.rb:244:in `each_top_level_statement'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/irb/ruby-lex.rb:230:in `loop'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/irb/ruby-lex.rb:230:in `each_top_level_statement'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/irb/ruby-lex.rb:229:in `catch'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/irb/ruby-lex.rb:229:in `each_top_level_statement'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/irb.rb:146:in `eval_input'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/irb.rb:70:in `start'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/irb.rb:69:in `catch'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/irb.rb:69:in `start'
 from /usr/bin/irb:13
Maybe IRB bug!!
>> Time.at(67767976233550799)
=> Tue Dec 31 23:59:59 -0500 2147483647
>> Time.at(67767976233550800)
=> Wed Jan 01 00:00:00 -0500 -2147483648

These aren't the limits in JRuby, as you can see here:


$ jruby -version
jruby 1.4.0 (ruby 1.8.7 patchlevel 174) (2009-11-02 69fbfa3) (Java HotSpot(TM) 64-Bit Server VM 1.6.0_20) [x86_64-java]
$ jirb
irb(main):001:0> Time.at(-67768040609636400)
=> Wed Jan 06 09:43:26 -0500 190734449
irb(main):002:0> Time.at(-67768040609636401)
=> Wed Jan 06 09:43:25 -0500 190734449
irb(main):003:0> Time.at(67768036191694799)
=> Fri Dec 25 14:20:30 -0456 -190730650
irb(main):004:0> Time.at(67768036191694800)
=> Fri Dec 25 14:20:31 -0456 -190730650

Tested in MacRuby, and it is like the regular Ruby MRI:


$ macruby -version
MacRuby version 0.6 (ruby 1.9.0) [universal-darwin10.0, x86_64]
$ macirb
irb(main):001:0> Time.at(67767976233550799)
=> 2147483647-12-31 23:59:59 -0500
irb(main):002:0> Time.at(67767976233550800)
=> -2147483648-01-01 00:00:00 -0500
irb(main):003:0> Time.at(-67768040609636400)
=> -2147481748-01-01 00:00:00 -0500
irb(main):004:0> Time.at(-67768040609636401)
ArgumentError: localtime error

irb(main):005:0> Time.at(67768036191694799)
=> -2147481749-12-31 23:59:59 -0500
irb(main):006:0> Time.at(67768036191694800)
ArgumentError: localtime error

However, these numbers seem to differ in 32-bit environments. In codepad, all attempts produce the following error. Steven Hazel of codepad.org stated that this could be a 32-bit OS-specific issue:


Line 1:in `at': bignum too big to convert into `long' (RangeError)
>  from t.rb:1

nib Method to Return nil if blank? in Ruby

2010-07-29T17:09:00.002-04:00

There are a number of solutions out there for returning nil if a variable is nil or blank? in Ruby but I came up with this which I think is easiest. Throw it into some .rb file under lib in your Rails project or just add it to a script.


class Object
  
  # Return nil if object.self.blank?, otherwise return object.self
  #
  # Author::  Gary S. Weaver
  
  def nib
    self.blank? ? nil : self
  end

end

Example usage:


>> 'John Doe'.nib
=> "John Doe"
>> ' '.nib
=> nil
>> ''.nib
=> nil
>> nil.nib
=> nil
>> 0.nib
=> 0

This allows you to do the following: (for example, in a view)


@director_name.nib || 'no name specified'

Note: nib stands for "nil if blank?".

Google's Server Error/Failure Page

2010-07-27T15:11:00.004-04:00

Something I don't see very often...

Notice the inline styling and no use of graphics; it's self-contained as you'd expect.


<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>502 Server Error</title>
<style><!--
body {font-family: arial,sans-serif}
div.nav {margin-top: 1ex}
div.nav A {font-size: 10pt; font-family: arial,sans-serif}
span.nav {font-size: 10pt; font-family: arial,sans-serif; font-weight: bold}
div.nav A,span.big {font-size: 12pt; color: #0000cc}
div.nav A {font-size: 10pt; color: black}
A.l:link {color: #6f6f6f}
A.u:link {color: green}
//--></style>
<script><!--
var rc=502;
//-->
</script>
</head>
<body text=#000000 bgcolor=#ffffff>
<table border=0 cellpadding=2 cellspacing=0 width=100%><tr><td rowspan=3 width=1% nowrap>
<b><font face=times color=#0039b6 size=10>G</font><font face=times color=#c41200 size=10>o</font><font face=times color=#f3c518 size=10>o</font><font face=times color=#0039b6 size=10>g</font><font face=times color=#30a72f size=10>l</font><font face=times color=#c41200 size=10>e</font>&nbsp;&nbsp;</b>

<td>&nbsp;</td></tr>
<tr><td bgcolor="#3366cc"><font face=arial,sans-serif color="#ffffff"><b>Error</b></td></tr>
<tr><td>&nbsp;</td></tr></table>
<blockquote>
<H1>Server Error</H1>
The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.

<p>
</blockquote>
<table width=100% cellpadding=0 cellspacing=0><tr><td bgcolor="#3366cc"><img alt="" width=1 height=4></td></tr></table>
</body></html>

Ruby Operator Precedence

2010-07-07T16:25:00.002-04:00

Ruby is about simplicity and clarity, but if you don't understand operator precedence, and think that "||" is the same as "or", it will eventually bite you.

Take for example the case of an orange:


class Orange
 def sweet?
   true
 end

 def sour?
   false
 end
end

Very simple, right? You can do things such as:


orange = Orange.new
puts "This orange is sweet" if orange.sweet?
puts "This orange not sour" unless orange.sour?

But, lets try to make a method to test whether an orange is sweet or sour.


def sweet_or_sour(orange)
 answer = orange.sour? or orange.sweet?
 puts "The orange is either sweet or sour: #{answer}"
end

Now call it:


orange = Orange.new
sweet_or_sour(orange)

What? The answer should be true! Why did this happen?

The answer is that the "or" has lower precedence than the assignment operator "=". The same is true for "and" vs. "&&" ("and" has lower precedence than "="). "not" is also lower precedence than "=", but "not" is higher than "or" or "and".

For more info, see Table 18.4 of The Pragmatic Programmer's Guide to Programming Ruby.

The single-table inheritance mechanism failed to locate the subclass: 'regular'

2010-07-01T10:27:00.001-04:00

If you get the following when trying to use ActiveRecord to view legacy tables in a database:


/Library/Ruby/Gems/1.8/gems/activerecord-2.3.8/lib/active_record/base.rb:1667:in `instantiate': The single-table inheritance mechanism failed to locate the subclass: 'regular'. This error is raised because the column 'type' is reserved for storing the class in case of inheritance. Please rename this column if you didn't intend it to be used for storing the inheritance class or overwrite NameOfYourModel.inheritance_column to use another column for that information. (ActiveRecord::SubclassNotFound)

Add this before your model definitions:


ActiveRecord::Base.inheritance_column = "activerecordtype"

Atlassian 20% Time and FedEx Day

2010-07-01T10:07:00.003-04:00

This is a great presentation by Atlassian on their 20% Time:
* Innovation@Atlassian: FedEx Day and 20% Time - Atlassian Summit: Charlie Talk

Summary:

Rules:

Is 20% of the development time that would have been spent, not support time, vacation, or conferences.
Engineers should work on something at least related to the company, the company's products, or open-source components used by the company. (This doesn't need to be enforced very often.)
To continue on the project and continue getting support: (This encourages projects to "fail early". But failure is not a problem. Innovation is about taking risks.)
- If have worked on it 5 working days - must have 3 other engineers to say I think that is a good project and you should continue working on it. This is done via periodic reviews.
- If have worked on it 3 weeks - must have company founder's approval to continue working on it.

Scheduling:

Easier on larger teams (for example, a developer that wants to take off a week to work on something).
Timekeeping - hard to get developers to do that. Someone just has to chase them with a whip.

Challenge:

Merging functionality with products, having to support life of project.
It will affect schedule of products because of that, but is providing features.

Sorts of projects usually worked on (up to developer to decide):

Not big things.
Things/bugs that annoy people.

More info and examples of things created with 20%:
Things tagged as "20 percent time" on Atlassian's developer blog

Fluid Textboxes with CSS

2010-06-24T13:27:00.000-04:00

If you have a fluid layout, you probably want fluid textboxes also. You'd think it would just be the default these days, but it isn't.


textarea {
  resize:both;
  width:100%;
}

Model to CSV in Rails

2010-06-24T12:14:00.006-04:00

If you have to produce a report/CSV of data you're storing with Rails, you may eventually be required to create a CSV report of those model attributes in CSV format.

A generic CSV implementation to convert an ActiveRecord model array to CSV may work at first:


require 'fastercsv'
...
def to_csv( records )
  csv_string = FasterCSV.generate do |csv|
    did_header = false      
    records.each do |record|
      unless did_header
        csv << record.attribute_names.collect! {|attr_name| attr_name.titleize} 
        did_header = true
      end        
      values = []
      record.attribute_names.each do |attr_name|
        value = record[attr_name]
        value = value.join(', ') if value.respond_to?(:join)
        value = value.strftime('%Y-%m-%d') if value.respond_to?(:strftime)
        values.push value
      end
      csv << values      
    end
  end
  csv_string
end
...

It would be great if something like that would suffice!

But eventually, they may ask for something like the report columns being in the order they were gathered in and to include additional fields, etc. At this point, you may end up defining all of the fields (and possibly their headers) in arrays in one of your classes. Then you end up with something similar to:

require 'fastercsv'
...
def to_csv( records )
  csv_string = FasterCSV.generate do |csv|
    csv << SomeModel.csv_headers
    records.each do |record|
      csv << record.csv_data
    end
  end
  csv_string
end
...

And in the Model you may have something like:


def self.csv_headers
  [
    "Name",
    "Date",
    "Favorite Albums"
  ]
end

def csv_data
  [ 
    self.some_associated_model.try(:display_name),
    self.some_date.try(:strftime, '%m-%d-%Y'),
    self.some_array.try(:join, ', ')
  ]
end

Of course, you are probably going to be doing quite a bit of copying and pasting at this point, which is prone to error. I find it helpful to clean up the datasets using find/replace in a text editor and then compare side-by-side in Excel or a similar spreadsheet utility (sort by name on each column individually just to make sure you didn't miss any fields, etc.). If you have regexp functionality in an editor like TextMate, use it. It is a great time to hone your RegEx skills. For example, to convert some_attribute_name to "Some Attribute Name" may partially involve capitalizing each word after the other substitutions, which you can do via a find:


(\w+)

and replace with:


\u$1

then check "Regular Expression", then do Replace and Find until you have converted the set of labels. Then go back and make corrections (like changing "Us" to "US" and "By" to "by", etc.).

Git Branch from Branch/Remote Branch and Making the New Branch Tracking Remote

2010-06-14T12:57:00.002-04:00

It is an easy enough concept: You have a remote branch you created and you want to branch off of it into a new branch that is also remote and tracking. However, it was hard to find an example of this anywhere that contained all of the steps involved, so here they are. Thanks to Paul D. for his help and to Richard for support with this.

Let's assume you already have a remote version branch and that your remote repo is "origin". Creating a branch off of master remotely and tracking it is described elsewhere, and I have a gbranch function in my git-scripts project for that purpose. But what if you want to branch off of that remote branch?

Ok, DON'T CHANGE THIS --no-track to -t/--track thinking you are smart. The goal here is to create a local copy of the remote 4.0 branch, not to create a local copy of it and track the remote existing branch. We'll add correct tracking in a minute!


git co --no-track -b new_branch_name origin/existing_remote_branch_to_branch_from

Now push that new branch to the remote repository:


git push origin new_branch_name

Finally, add tracking to the new_branch. If you have git 1.7.0 or later (do a git --version) then you can do this:


git branch --set-upstream new_branch_name origin/new_branch_name

If you don't have git 1.7.0 or later, you can edit .git/config and add something like:


[branch "new_branch_name"]
        remote = origin
        merge = refs/heads/new_branch_name

Form with Multiple-selection Select Stored in Rails Comma-delimited Model Attribute

2010-06-09T15:23:00.011-04:00

Here is an example for how you can do multiple-selections in a Rails form and store as a comma-delimited single field. This solution is based off of the one by smallbeer and Bill Posters, but in addition it sets the selections.


<p>
  <script src="http://jqueryjs.googlecode.com/files/jquery-1.3.2.min.js" type="text/javascript"></script>  
  <script>
$.noConflict();
 jQuery(document).ready(function($) {

  function setHandlers(e) {
   $('select[id^=select_favorite_foods]').change(updateValue);
  }

  function updateValue() {
   var select_values = [];
   $('select[id^=select_favorite_foods]').each(function(e) {
    if ($(this).val()) select_values.push($(this).val());
   });
   $('#name_of_your_model_favorite_foods').val(select_values.join(','));
  }
  
  function initSelection() {
    vals = $('#name_of_your_model_favorite_foods').val().split(',');
    var select_values = [];
    for ( i = 0; i < vals.length; i++) {
      if (vals[i]) select_values.push(vals[i]);
    }
    $('select[id^=select_favorite_foods]').val(select_values);
  }

  $(document).ready(function() {
   initSelection();
   setHandlers();
   updateValue(); 
  });
 });
  </script>
  <%= f.label :favorite_foods, "Select your favorite foods:" %><br />
  <%= f.hidden_field :favorite_foods %>
  <select id="select_favorite_foods" multiple="true" size="3">
    <option value="Orange">Orange</option>
    <option value="Banana">Banana</option>
    <option value="Grapefruit">Grapefruit</option>
  </select>
</p>

Messy, but it is just an example! However, for Rails, if you don't mind storing the value in the DB as YAML, that is a little easier. Also, this same technique with a few mods could work for any HTML form.

(Thanks to Jeremy who provided ideas when I was trying to work through the initSelection method.)

Formatting Java in TextMate

2010-06-07T15:36:00.000-04:00

If you need to reformat Java code in TextMate (in OS X), a simple way to do it is via:


command+a
opt+command+[

I have this note sitting on my desk as a reminder, so I thought it would be good to share. This is from Vinod in this post on stack overflow.

Grammar Checking in OS X 10.6/Snow Leopard

2010-06-03T13:21:00.017-04:00

There are so many features in OS X 10.6.3 that I like, but there is one that is extremely deficient: grammar checking. I would much rather have Mail.app correct my grammar mercilessly. Instead, it lets grammar pass that would make almost anyone with any knowledge of English grammar guffaw. (Sorry for using the word "guffaw". Guffawing is just something I imagine a grammarian would do.)

Having something is sometimes better than nothing. For example, as noted in a post that describes how to turn OS X grammar checking on, it knows "With regards to" is incorrect and has a tip that states "Is this an error for 'With regard to'? Try 'Concerning' or 'About' instead."

However, the appearance that it is checking your grammar is misleading. If I type the following, "With regard to trees, I like apple yet than this is not a grammatically correct sentence come on please please please you need to at least notice some grammatical mistake here.", it checks out. Huh? If you want a real grammar checker in OS X, you might just buy Grammarian or a similar product.

How to Turn on Ojdbc Driver Debug Logging

2010-05-21T16:18:00.002-04:00

It had been a while since I turned on ojdbc trace logging, so I had to look it up. Found a great article called Turning on Oracle Driver Tracing by Kualigan and added my notes below. I didn't use the Sun Java Logging API part of it.

If you need to turn on Oracle JDBC driver debug or trace logging, you first need to install the ojdbc driver "g" jar provided by Oracle (for example, ojdbc14_g.jar).

If using Maven 2 and you don't want to make changes to your pom.xml(s), you could just swap out the version in your local repo with the debug version, but that may cause you later to accidentally include the debug version in another project, so I just changed the version in my pom.xml to a version I knew I didn't have called "9.2.0.8debug" (because I downloaded the ojdbc v9.2.0.8 debug driver).


        <dependency>
            <groupId>com.oracle</groupId>
            <artifactId>ojdbc14</artifactId>
            <!--<version>10.2.0.4.0</version>-->
            <version>9.2.0.8debug</version>
            <exclusions>
              <exclusion>
                <groupId>org.apache.tomcat</groupId>
                <artifactId>catalina</artifactId>
              </exclusion>
            </exclusions>
        </dependency>

Then when I rebuilt, I used the command Maven 2 told me to use to manually install that jar.


mvn install:install-file -DgroupId=com.oracle -DartifactId=ojdbc14 -Dversion=9.2.0.8debug -Dpackaging=jar -Dfile=/Users/myuser/Downloads/ojdbc14_g.jar

Then if you are trying to trace in a functional test run by Maven, you modify pom.xml's surefire plugin config in its build plugins to turn on trace logging and specify a Java logging API config file:


            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <configuration>
                    <reportFormat>brief</reportFormat>
                    <useFile>false</useFile>
                    <systemProperties>
                        <property>
                            <name>oracle.jdbc.Trace</name>
                            <value>true</value>
                        </property>
                    </systemProperties>
                </configuration>
            </plugin>

More info:
* Turning on Oracle Driver Tracing

Zip or War All Directories in Current Directory into Individual Zip or War Files

2010-05-20T13:58:00.001-04:00

If you need to zip or war all of the directories only in the current directory into their own zip or war files, here are a few approaches:

To zip all of the directories only in the current directory into individual zip files and retain the directory name in the zipped files so that when they expand they expand back into that directory, just use:


find * -prune -type d -exec zip -r "{}.zip" "{}" \;

But if you want to zip or war each so that you change directories (you change the current directory/working directory such that when the zip expands, it expands the files and directories of that directory into the current directory, as you would expect in a war file), you can't just use "cd" in the exec, and the unix zip command doesn't appear to let you change the working/current directory. The easiest thing to do is to have a separate shell script that changes the current working directory and does the zip.

Create a file called warit.sh or zipit.sh and put it on the path.

zipit.sh:


#!/bin/bash
set -e
cd $1
zip -r "../$1.zip" .

warit.sh:


#!/bin/bash
set -e
cd $1
jar cvf "../$1.war" .

then just:


find * -prune -type d -exec zipit.sh {} \;


find * -prune -type d -exec warit.sh {} \;

and it will cd into each of the directories, zip all that up, and place the resulting zip or war into the parent directory. This is basically a hack for zip not taking an argument to specify the working directory, and because cd has difficulties running within find's exec. I fought this a good while and would appreciate if someone has a more elegant solution or could fight to allow the OS X 10.6.3 zip command to specify the current working directory.

Instead of "find * -prune -type d -exec ...", you can also use "find . -d 1 -type d -exec ...".

Unplutofy Cleans web.xml and portlet*.tlds from Your Portlet Wars

2010-05-20T11:25:00.005-04:00

Just wrote Unplutofy, which includes an Ant task, Maven plugin, Command-line tool, and Java API to clean Apache Pluto additions to Portlet wars. What it does is fairly simple, but there wasn't anything in Pluto or uPortal that did this already.

Why do this?

Sometimes you may want to redeploy a portlet war that has already been deployed into a different version of a portal that has a different Pluto version. Since Pluto can be used (in uPortal via deployPortletApp, at least in past versions) to just tack on servlet and servlet-mapping tags to the web.xml without making sure that the old ones from past versions (or even current versions) are removed (see PLUTO-576), it is helpful to know there is a way to revert the wars to the un-plutofied state.

View and Graph Git Repositories in OS X

2010-05-18T09:15:00.001-04:00

GitX is a great tool for visualizing git repositories in Mac OS X. Just unzip the app, drag it into your Applications, start it, and got to GitX menu -> Enable Terminal Usage... That will put it into your bin directory so that you can just type:


gitx

while in a git repo directory, and it will popup with a visual graph of the repository.

Pretty neat, and pretty slick, too!

For more info, see:
* http://www.gitready.com/intermediate/2009/01/13/visualizing-your-repo.html

How to Shibbolize uPortal 3.2.1

2010-05-14T14:04:00.011-04:00

This assumes you've already setup the Java 6 SDK, uPortal 3.2.1, Shibboleth, and know that Shibboleth is sending the username to Tomcat via a REMOTE_USER header. It also assumes you have added the tomcatAuthentication="false" to all relevant connectors in Tomcat 6's server.xml. For assistance with these tasks, go to the official site and the shibboleth-users mailing list.

Before you start, you'll probably want to add an admin user with the same ID as that which will come through via REMOTE_USER from Shibboleth. If you don't, you can just change "admin" to your username in UP_USER.USER_NAME, UP_PERSON_DIR.USER_NAME, and in UP_GROUP_MEMBERSHIP.MEMBER_KEY.

After doing all that, here is one way to Shibbolize uPortal v3.2.1. This assumes you don't need any authN except Shibboleth:

Edit WEB-INF/classes/layout/theme/universality/components.xsl


 <xsl:template name="login">
    <div id="portalLogin" class="fl-widget">
      <div class="fl-widget-inner">
        <form method="post" action="Login">
            <fieldset id="portalLogin">
                <ul><li><input class="uportal-button" name="Login" value="Login" type="submit" /></li></ul>
                <xsl:apply-templates/>
            </fieldset>
        </form>
      </div>
    </div>                             
  </xsl:template>

This assumes that you have configured Shibboleth to guard Login under uPortal (example: https://employees.acme.com/uPortal/Login). It also assumes you are using that theme. For mobile support, make a similar change to WEB-INF/classes/layout/theme/muniversality/components.xsl.

I also modified WEB-INF/classes/org/jasig/portal/channels/CLogin/html.xsl with the following. This is the template for local login and it shouldn't be used if you make the change above.


   <xsl:template match="login-status">
        <form method="post" action="Login">
            <fieldset id="portalLogin">
                <ul><li><input class="uportal-button" name="Login" value="Login" type="submit" /></li></ul>
                <xsl:apply-templates/>
            </fieldset>
        </form>  
    </xsl:template>

Edit WEB-INF/classes/properties/contexts/userContext.xml


<!--<bean id="personManager" class="org.jasig.portal.security.provider.SimplePersonManager"/> -->
<bean id="personManager" class="org.jasig.portal.security.provider.RemoteUserPersonManager"/>

Edit WEB-INF/classes/properties/security.properties


## This is the factory that supplies the concrete authentication class
root=org.jasig.portal.security.provider.UnionSecurityContextFactory
root.remoteuser=org.jasig.portal.security.provider.RemoteUserSecurityContextFactory
#root.cas=org.jasig.portal.security.provider.cas.CasAssertionSecurityContextFactory
#root.simple=org.jasig.portal.security.provider.SimpleSecurityContextFactory

...

## Answers where the user will be redirected when log out occurs. Each security context can have one.
## (See comments in the LogoutServlet class)
## It would be better to escape the value of the url parameter, but since there are no parameters on the
## unescaped URL and since there are no further parameters on the logout URL, this does work.
#logoutRedirect.root=http://localhost:8080/cas/logout?url=http://localhost:8080/uPortal/Login
logoutRedirect.root=/Shibboleth.sso/Logout

There is a better way of doing this that would allow it to be an option along with local login and CAS, and I offered to provide such a patch to uPortal, but there didn't seem to be any interest, so I didn't spend additional time on it.

How to Fix Net::HTTPBadResponse: wrong status line: "" in Ruby

2010-05-03T16:23:00.007-04:00

Using Ruby 1.8.7:


$ ruby -version
ruby 1.8.7 (2008-08-11 patchlevel 72) [universal-darwin10.0

If I do the following, you can see where it might be confusing that the error has nothing to do with me not specifying that I wanted to use HTTPS vs HTTP:


$ irb
>> require 'net/http'
=> true
>> Net::HTTP.get_response(URI.parse('http://www.google.com/'))
=> #<Net::HTTPOK 200 OK readbody=true>
>> Net::HTTP.get_response(URI.parse('https://www.verisign.com/'))
Net::HTTPBadResponse: wrong status line: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">"
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:2022:in `read_status_line'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:2009:in `read_new'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:1050:in `request'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:948:in `request_get'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:380:in `get_response'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:543:in `start'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:379:in `get_response'
 from (irb):3

But if I change the code to specify I want to use HTTPS:


$ irb
>> require 'net/http'
=> true
>> require 'net/https'
=> true
>> url = URI.parse('https://www.verisign.com/')
=> #<URI::HTTPS:0x101200588 URL:https://www.verisign.com/>
>> http = Net::HTTP.new(url.host, url.port)
=> #<Net::HTTP www.verisign.com:443 open=false>
>> http.use_ssl = true
=> true
>> request = Net::HTTP::Get.new(url.path)
=> #<Net::HTTP::Get GET>
>> response = http.start {|http| http.request(request) }
warning: peer certificate won't be verified in this SSL session
=> #<Net::HTTPOK 200 OK readbody=true>

all is good!

This came about because an example for an HTTP GET in the Ruby standard documentation doesn't mention HTTPS, and it is only natural that at some point you'll try to use that same code to get something via SSL/HTTPS, and it will fail with a similar error to the above.

Here is an imperfect generic implementation of GET utility method based on the original:


  # copied from http://ruby-doc.org/stdlib/libdoc/net/http/rdoc/classes/Net/HTTP.html then modified to support SSL
  def self.fetch(uri_str, limit = 10)
    # You should choose better exception.
    raise ArgumentError, 'HTTP redirect too deep' if limit == 0
    
    url = URI.parse(uri_str)      
    http = Net::HTTP.new(url.host, url.port)
    http.use_ssl = (url.scheme == 'https')
    request = Net::HTTP::Get.new(url.path)
    response = http.start {|http| http.request(request) }
      
    case response
    when Net::HTTPSuccess     then response
    when Net::HTTPRedirection then fetch(response['location'], limit - 1)
    else
      response.error!
    end
  end

OS X 10.6.2/Snow Leopard Login Issues

2010-05-03T09:26:00.017-04:00

I tried to login to OS X 10.6.2/Snow Leopard multiple times this morning into the black background login from the screensaver and got rejected every time. I rebooted, then had to login both in the main boot login screen and the black login screen. I usually would only have to login once via the main boot login after a restart.

If you have this trouble also, I suggest holding down the power button until the Mac goes off, press it again after it is off to start it back up, then login (as many times as necessary) to get in.

My first thought was that it was a virus/trojan/worm/some kind of malware, or I got hacked. I'm still not ruling those out, but it is suspicious that others on Twitter are reporting similar issues. Wondering if it is related to a recent update from Apple?

Getting Maven to Work in Hudson

2010-04-29T12:21:00.016-04:00

We were having some trouble today getting a Maven 2 project to build in Hudson v1.353. Specifically, we were getting an 'Exception in thread "main" java.net.ConnectException: Connection refused'.

We added:


-Djava.awt.headless=true

to MAVEN_OPTS in the hudson configuration page (per Martijn Dashorst and mbrictson's posts) and this issue seems to have gone away. Thanks much to Martijn and mbrictson for sharing this solution!

It is strange that after setting this and then removing it, projects continue to build correctly, even if MAVEN_OPTS in each individual project (in Advanced under Build section of your projects Configure page in Hudson) does not contain this parameter. It also continued to work after Hudson was restarted.

Note: I had previously installed Maven to a local spot manually instead of via Hudson. I tried to let Hudson automatically install it, but it had some wierd errors on build. First I had some rights-related issues that I fixed, but then I ran into a SocketTimeoutException and just decided install Maven 2 myself in different directory than where Hudson puts it.

Bad Existing Git Repo Causes Subsequent Svn to Git Conversion Error

2010-04-28T15:45:00.002-04:00

If you get the following error:


$ git svn clone svn+ssh://userid@some.host/path/to/svn/project/dir --no-metadata -T trunk /path/to/local/git/repo/project/dir
W: Ignoring error from SVN, path probably does not exist: (160013): Filesystem has no item: File not found: revision 8481, path 'misspelled/or/invalid/path/to/svn/project/dir'
W: Do not be alarmed at the above message git-svn is just searching aggressively for old history.
This may take a while on large repositories

try removing the directory of the git repo that you are trying to migrate into from svn, or just move it out of the way:


mv /path/to/local/git/repo/project/dir /path/to/local/git/repo/project/dir.bak

It doesn't appear to like having a poorly migrated git repo already there.

Misspelled Hostname in SVN URL Causes Wierd Error When Converting Subversion to Git

2010-04-28T15:14:00.007-04:00

Got the following error when trying to convert an svn project to a git repo:


$ git svn clone svn+ssh://userid@some.host/path/to/svn/project/dir --no-metadata -T trunk /path/to/local/git/repo/project/dir
Use of uninitialized value in concatenation (.) or string at /opt/local/lib/perl5/vendor_perl/5.8.9/darwin-2level/SVN/Core.pm line 584.
Network connection closed unexpectedly:  at /opt/local/libexec/git-core/git-svn line 1888

I determined that the host name in the svn URL was incorrect. The "Use of uninitialized value in concatenation (.) or string at /opt/local/lib/perl5/vendor_perl/5.8.9/darwin-2level/SVN/Core.pm line 584." didn't help much. The hint here was "Network connection closed unexpectedly" after which I just happened to notice the bad hostname. Actually, I think it was a bad character inserted because I copied/pasted of a number of commands at once into terminal.app, which tends to cause issues. Hope this helps someone else.

TextMate Slowness with Ruby on Rails Language Grammar

2010-04-27T10:50:00.017-04:00

When editing an erb file, I noticed that when I escaped most of the EOL chars with <%# and put %> at beginning of the following lines, TextMate 1.5.9 (build 1510) got really slow; the beach ball just kept spinning and spinning in OS X 10.6.2 Snow Leopard, and it appeared to be locked-up/frozen/non-responsive.

I ended up fixing this by changing the language grammar at the bottom of the TextMate window to "HTML" instead of "Ruby on Rails" for .erb files.

Note: something else to look into when TextMate is slow is the number of files in the projects you have open. Try closing projects or reducing the number of files in the project if things are slow.

Modular Design vs. MVC

2010-04-26T13:55:00.026-04:00

Usually the implementation of the View in MVC is something that is so geared towards presentation or the final display of the data that reuse by other components (those not interested in the presentation, but only interested in the data produced for the presentation) are not considered.

Think of just about any standard framework used to help implement the view for a web application in any programming language. Do you have an easy way to just call that code from the controller and get its response for reuse somewhere else in the application? Probably not. So, why would you want to?

In an Rails app + XML/JSON/JSONP service that we had developed, the first version of the app/service did the XML/JSON/JSONP generation outside of the view code. For example, the to_xml implementation was just another method, so it would have been simple enough to reuse it almost anywhere within the app.

However, in the second version of the app/service, this was changed so that there was a view to generate the XML. Considering XML to be a view just like anything else seems at first to a good idea. But it makes it more complex now to do the equivalent of to_xml if you need it elsewhere in the app. This seems to have somewhat sacrificed Modular Design for the traditional push of presentation-related code into the view layer.

MVC would at first appear just to be a subset of Modular Design. But the problem is that when you say (just as an example), "This functionality is just for the view, so I don't really care if you want to use it for anything else, because my job is just to provide the last step prior to presentation," you are probably sacrificing some degree of modularity and potential reuse.

For webapps, there are obvious benefits for using MVC, and odds are the framework you are using promotes MVC. However, it makes sense to at least consider Modular Design principles when coding bits of functionality that could be reused.

git-core +svn MacPorts Install in Snow Leopard

2010-04-23T13:08:00.003-04:00

Paul Downman has a great article on converting svn to git repositories, but upon trying to run the first command in OS X 10.6.2:


sudo port install git-core +svn

I got:


...
--->  Building cyrus-sasl2
Error: Target org.macports.build returned: shell command " cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_security_cyrus-sasl2/work/cyrus-sasl-2.1.23" && /usr/bin/make -j3 all " returned error 2
Command output:  fi
if /bin/sh ../libtool --mode=compile /usr/bin/gcc-4.2 -DHAVE_CONFIG_H -I. -I. -I.. -I../include -I../include   -I/opt/local/include -I/opt/local/include  -Wall -W -O2 -arch x86_64 -MT db_ndbm.lo -MD -MP -MF ".deps/db_ndbm.Tpo" \
   -c -o db_ndbm.lo `test -f 'db_ndbm.c' || echo './'`db_ndbm.c; \
 then mv -f ".deps/db_ndbm.Tpo" ".deps/db_ndbm.Plo"; \
 else rm -f ".deps/db_ndbm.Tpo"; exit 1; \
 fi
if /usr/bin/gcc-4.2 -DHAVE_CONFIG_H -I. -I. -I.. -I../include -I../include   -I/opt/local/include -I/opt/local/include  -Wall -W -O2 -arch x86_64 -MT db_ndbm.o -MD -MP -MF ".deps/db_ndbm.Tpo" \
   -c -o db_ndbm.o `test -f 'db_ndbm.c' || echo './'`db_ndbm.c; \
 then mv -f ".deps/db_ndbm.Tpo" ".deps/db_ndbm.Po"; \
 else rm -f ".deps/db_ndbm.Tpo"; exit 1; \
 fi
if /usr/bin/gcc-4.2 -DHAVE_CONFIG_H -I. -I. -I.. -I../include -I../include   -I/opt/local/include -I/opt/local/include  -Wall -W -O2 -arch x86_64 -MT allockey.o -MD -MP -MF ".deps/allockey.Tpo" \
   -c -o allockey.o `test -f 'allockey.c' || echo './'`allockey.c; \
 then mv -f ".deps/allockey.Tpo" ".deps/allockey.Po"; \
 else rm -f ".deps/allockey.Tpo"; exit 1; \
 fi
/usr/bin/gcc-4.2 -DHAVE_CONFIG_H -I. -I. -I.. -I../include -I../include -I/opt/local/include -I/opt/local/include -Wall -W -O2 -arch x86_64 -MT allockey.lo -MD -MP -MF .deps/allockey.Tpo -c allockey.c  -fno-common -DPIC -o allockey.lo
/usr/bin/gcc-4.2 -DHAVE_CONFIG_H -I. -I. -I.. -I../include -I../include -I/opt/local/include -I/opt/local/include -Wall -W -O2 -arch x86_64 -MT db_ndbm.lo -MD -MP -MF .deps/db_ndbm.Tpo -c db_ndbm.c  -fno-common -DPIC -o db_ndbm.lo
allockey.c: In function '_sasldb_putsecret':
allockey.c:191: warning: pointer targets in passing argument 6 of '_sasldb_putdata' differ in signedness
allockey.c: In function '__sasldb_internal_list':
allockey.c:197: warning: unused parameter 'rock'
allockey.c: In function '_sasldb_putsecret':
allockey.c:191: warning: pointer targets in passing argument 6 of '_sasldb_putdata' differ in signedness
allockey.c: In function '__sasldb_internal_list':
allockey.c:197: warning: unused parameter 'rock'
mv: rename .deps/allockey.Tpo to .deps/allockey.Plo: No such file or directory
make[2]: *** [allockey.lo] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2

Error: The following dependencies failed to build: p5-svn-simple subversion-perlbindings cyrus-sasl2 neon serf subversion p5-term-readkey
Error: Status 1 encountered during processing.
Before reporting a bug, first run the command again with the -d flag to get complete output.

After getting this I first tried to clean the dependencies that failed to build:


sudo port clean p5-svn-simple subversion-perlbindings cyrus-sasl2 neon serf subversion p5-term-readkey

and then did "sudo port install git-core +svn" again, and it failed. Following a lead on something else I saw, I tried:


sudo port install cyrus-sasl2 +universal

and got:


--->  Computing dependencies for cyrus-sasl2
--->  Fetching cyrus-sasl2
--->  Verifying checksum(s) for cyrus-sasl2
--->  Extracting cyrus-sasl2
--->  Applying patches to cyrus-sasl2
--->  Configuring cyrus-sasl2
Error: You cannot install cyrus-sasl2 for the architecture(s) x86_64 i386 because
Error: its dependency openssl only contains the architecture(s) x86_64.
Error: 
Error: Try rebuilding openssl (and all its dependencies) with
Error: the +universal variant by running
Error: 
Error:     sudo port upgrade --enforce-variants openssl +universal
Error: 
Error: Target org.macports.configure returned: incompatible architectures in dependencies
Error: Status 1 encountered during processing.
Before reporting a bug, first run the command again with the -d flag to get complete output.

So then I did:


sudo port upgrade --enforce-variants openssl +universal
sudo port install cyrus-sasl2 +universal
sudo port install git-core +svn

and it installed.

Nil Model Attributes and Lazy-loading

2010-04-15T09:43:00.019-04:00

I was working on a project where there was a method in a model that I defined with references to ActiveRecord model attributes like @attribute_name instead of via the accessors like self.attribute_name.

In one route and in one controller, a method checking @attribute_name would return the desired instance of the object. In another route and another controller, it would return nil, even though that attribute was not nil when accessed by its views.

Paul D. mentioned that I should try using an accessor instead of direct attribute reference, and, sure enough, this issue was caused by trying to access an attribute directly that ActiveRecord would have lazy-loaded when retrieved via accessor.

As Paul said, unless the attributes you are working with are ones that you created instead of ActiveRecord's, it is a better idea to use the accessors in your models.

Push CMSs and Version Control

2010-04-09T12:03:00.013-04:00

If you are choosing a CMS, you may want to look into whether it gives you a clean and obvious way to view differences between the managed content and the published/served content. This is not often a straightforward task. In fact, some CMSs have features that make this a nearly impossible task, because the content you view in the CMS does not have a 1:1 relationship with the published files in the case of a push CMS (possibly because there are transforms during the publish, etc.). Having audit history of changes made just within the CMS or just being able to view changes between versions of content in the CMS might be nice, but if it doesn't take into account what was actually published or is still viewable by the user or another application, those alone may not be sufficient.

A few examples of things that could go wrong if you don't have a good handle on what resides in the publish destination:

Users and applications can continue to access old content via deep-links.
You might not be aware of content that was corrupted on/after publish or was only partially-published.

For this reason, I'm a much bigger fan these days of the concept of a pull CMS (a CMS that serves content) as they seem to do a better job with "What you see is what you get". Caching and copying static rendered content from a pull CMS (a CMS that serves content) seems enough for most of the needs we'd have, even if it doesn't solve every problem.

Change sed's Pattern Delimiter Character

2010-04-08T07:51:00.015-04:00

sed has a neat feature as noted by mu where you can use any character to delimit the parts of the regular expression, which means you can control what characters would have to be escaped in your patterns.

Instead of using the normal forward-slash, you can use a pipe (|) or any character other than a backslash or newline character. Note that putting a backslash character before the delimiting character causes the character to be treated literally within the expression rather than as a delimiter.

For example, to replace all instances (because of the g) of the text "http://www.example.com/some/old/path" in old_file with "http://acme.com/some/new/path" and output to new_file, use:


sed -e 's|http://www.example.com/some/old/path|http://acme.com/some/new/path|g' old_file > new_file

Gotchas for Cross-domain Portlets via iFrames

2010-04-06T06:08:00.006-04:00

We explored the available (Google-able) cross-domain iframe solutions a few months ago to see if using iFrames for cross-domain apps we would want to run as portlets would be a good idea. But the problem is that in most cross-domain hacks the iframed webapp must initiate the resizing of its parent container via changing its target link. You can imagine how that would cause some issues:

The app you are trying to portlet-ize should not care whether it is in a portlet or not.
The app you are trying to portlet-ize should not have to do the resizing, because that requires that all actions in the app that could affect its size need to call the refresh methods that set the URL target of themselves that the parent is watching in order to relay the new height and/or width.

This isn't to say that iFrames won't work in portals. For example, I know that John King of Geneseo U. uses iFrames in their portal but they own all of the portlet-ed apps and they are all on the same domain. However, for those exploring cross-domain iframed webapps in their portal, be aware of these gotchas.

If you want to develop apps that would be hosted in a different domain than the portal, you might look into Jasig's WebProxyPortlet (to proxy the external app), rails-portlet (if developing Rails apps), portletbridge, Grails Portlets Plugin (if developing Grails apps), etc. Also, we're working on an idea in-house at the moment that involves a completely client-side based javascript/jQuery solution using jsonp to communicate with cross-domain services. Jeremy B. is doing the heavy-lifting with the jQuery UI and JSON usage and following that, I'm hoping to make a gem to handle the creation of that javascript/HTML- that way it will serve up the client-side code after which jsonp is used solely for additional actions.

However, if you don't have control over the external app you want to show up in the portal, then Jasig's WebProxyPortlet or Mikael Lammentausa's html2jsr286 might be options.

Hope this helps someone else.

How to Shibbolize uPortal 2.5.3.2

2010-04-02T12:34:00.012-04:00

We completed Shibbolization of our old version of uPortal (v2.5.3.2) today. Other than a good amount of work that various team members (Chris spent most of the time on it on the SP side) and I made to setup the SP and IdP's, the uPortal changes ended up not being so bad.

Modify WEB-INF/classes/org/jasig/portal/channels/CLogin/html.xsl:


<!-- ~ If user is not authenticated insert login form-->
    <!-- ~ -->
    <xsl:template match="login-status">
        <form method="post" action="Login">
            <fieldset id="portalLogin">
                <ul><li><input class="uportal-button" name="Login" value="Login" type="submit" /></li></ul>
                <xsl:apply-templates/>
            </fieldset>
        </form>  
    </xsl:template>

This assumes that you have configured Shibboleth to guard Login under uPortal (example: https://employees.acme.com/uPortal/Login).

Modify WEB-INF/classes/properties/portal.properties:


org.jasig.portal.security.PersonManagerFactory.implementation=org.jasig.portal.security.provider.RemoteUserPersonManager

Modify WEB-INF/classes/properties/security.properties:


root=org.jasig.portal.security.provider.RemoteUserSecurityContextFactory
logoutRedirect.root=/Shibboleth.sso/Logout

and modify the logoutRedirect.root if needed.

You can also disable/modify your logout link in (tomcat)/webapps/uPortal/WEB-INF/classes/org/jasig/portal/channels/CHeader/html.xsl and html_*.xsl in that dir.

Other resources:

Tags and Nested/Hierarchical/Categorical Organization

2010-03-30T13:21:00.024-04:00

Today we were looking at data for an app we have internally that was organizing tagged data in a hierarchy of categories (nested categories). I assume that in the original design, categories were thought to be something helpful to guide the user into a sensible set of tags which which to filter the data.

However, since the primary method of getting to the data was via categories, we are now seeing issues even with the small set of data that were caused because of the context of data being in a category, even if that data's tags did not adequately represent that context.

For example, there was a category called "Vegetarian Selection" and under that category were three categories: "Excellent", "Good", and "Limited". The corresponding tags should have been called excellent_vegetarian_selection, good_vegetarian_selection, and limited_vegetarian_selection. However, the "Vegetarian Selection" category was the only way to access data tagged with those tags, so that context made the tags: excellent, good, and limited seem reasonable to whoever set them up. Even though it is a minor change required in data (the tag names), this method of giving context to tags from what category they are in would seem problematic.

So how can you mix categories and tags and reduce the change of categorization giving the tag context? You could introduce the concept of "tag groups" and a sequential filtering by a member of these groups, similar to how Amazon does product browsing. For example:


tag_group: building

tags: green_center, west_building, ...

tag_group: cuisine

tags: cafe, tex-mex, vegetarian, ...

tag_group: vegetarian_selection

tags: limited_vegetarian_selection, good_vegetarian_selection, ...

and maybe an automatically available tag_group (?) called: misc that would contain everything else not in a group

So instead of a single call to get a single nested categories list, there would be two calls to:

return a list of tag_groups that have at least one tag that contained places associated with those tags, given the list of tags you are currently filtering by (which may be empty)
return a list of tags that have places associated with those tags, given the list of tags you are currently filtering by (which may be empty)

Using that solution, you could write something that keeps providing a list a tag_groups that you haven't defined a choice from yet. You keep choosing or you click on "all" to see a full list of places with the current list.

However, it is worth noting that categorization and hierarchy/nesting introduces a risk of associating the context of the category with tags, so be careful if you mix the two.

Capistrano Deploy Issue in 2.5.10

2010-03-25T12:04:00.007-04:00

When using set :keep_releases = (number) and Capistrano 2.5.10, it deleted the release it created on the server at the end of deployment. This would royally screw the deploy up, because in the end, the "current" symlink on the server would point to a non-existing release directory. I am very thankful that issue has been fixed between that version and current version, Capistrano 2.5.18.

Since there were no errors in the log and you don't really pay attention to the release numbers in the log since they are so long, this really wasn't obvious at first. I couldn't find the bug for it when searching, but perhaps you can. Update: Lee helped me find the bug (#88) as discussed in bug #155.

I figured this out when I was doing text-replacement to remove the release numbers from the old log and new to help with the diff. Doing that I noticed that in the end of the log of the deploy using Capistano 2.5.10 removed the release it had deployed (!) in the end of the log.


  * executing "rm -rf /path/to/project/releases/(number of release just deployed)"
    servers: ["the.server.name"]
    [the.server.name] executing command
    command finished

I'm fairly sure that this had to do with the set :keep_releases, 3 that we were using for this particular project.

So, the moral of the story is, if you are using Capistrano 2.5.10, be wary of :keep releases screwing up your deploy and leaving you with a current symlink pointed at a non-existant release dir. To fix, update capistrano via:


gem update capistrano

and redo your deployment.

LHC Operations

2010-03-19T14:19:00.002-04:00

Follow the LHC on Twitter via @CMSexperiment or via the LHC Operations status online. Links courtesy of this post on Slashdot. Took a screenshot of the LHC Operations status showing today's status:

Increase Side-by-side Diff Width for Terminal.app

2010-03-18T14:19:00.004-04:00

For some reason, I always lived with --side-by-side diff getting chopped-off. I've found a new friend with specifying -W number or --width=number:


diff -b -y -W 280 file1 file2

Or the more lengthy version:


diff --side-by-side --width=280 --ignore-space-change file1 file2

Yes, there are nice graphical ones, but this works for me.

Using sed to Escape Quotes

2010-03-18T13:14:00.010-04:00

Using sed to escape quotes in a piped in value:


$ echo "\"hello\"" | sed s/\"/\\\\\\\"/g
\"hello\"

But backticks make it more interesting. Setting result of that into variable, via backticks:


$ resp=`echo "\"hello\"" | sed s/\"/\\\\\\\\\"/g`; echo $resp
\"hello\"

Just for kicks, backticks within backticks (see Escaping Backticks Within Backticks):


$ resp2=`resp1=\`echo "\"hello\"" | sed s/\"/\\\\\\\\\\\\\\\\\"/g\`; echo $resp1`;echo $resp2
\"hello\"

One more level of backticks:


$ resp3=`resp2=\`resp1=\\\`echo "\"hello\"" | sed s/\"/\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/g\\\`; echo $resp1\`;echo $resp2`;echo $resp3
\"hello\"

And another:


$ resp4=`resp3=\`resp2=\\\`resp1=\\\\\\\`echo "\"hello\"" | sed s/\"/\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/g\\\\\\\`; echo $resp1\\\`;echo $resp2\`;echo $resp3`;echo $resp4
\"hello\"

(Note: it can seem forgiving at times on the number of backticks if you have a few too many.)

So, the number of backslashes required for the escaped slash replacement value is 7 if not within backticks. However, for each level of backticks, it requires 9,17,33,65,etc. backslashes.

Git Branching with Remote Tracking Made Easy

2010-03-17T11:33:00.003-04:00

I wanted to be able to just create or switch to a new or existing remote/local git branch with a single command, so that you don't have to worry about whether you or a teammate already created a remote branch for a specific development request. It just works. While it can be used for any branch name, if you are using Jira, Trac, or a similar ticketing system, it might be good to use gbranch (ticket name). Then because the branches it creates are remotely tracked, you can git push and git pull without specifying the origin.

To use it, create a file called gbranch.sh and chmod +x gbranch.sh and execute it in your start script (like .bash_profile, or whatever shell you use).

This script is also included in my git-scripts project.



# gbranch.sh
#
# Written by Gary S. Weaver
#
# Release: 2010-03-17
# This tracks a remote branch if it already exists, or
# makes a remotely tracking git branch if one doesn't exist, or
# just switches to the local branch by that name.

function gbranch () {
  branch=$1
  if test $# -eq 1
  then
    rbrch=`git branch -r`
    if [ "$?" -eq "0" ]
    then
      if [[ "$rbrch" =~ "origin/$branch" ]]
      then
        git checkout -b "$branch" --track "origin/$branch"
        if [ "$?" -ne "0" ]
        then
          git checkout "$branch"
        fi
      else
        git checkout -b "$branch"
        if [ "$?" -eq "0" ]
        then
          # The following lines track the remote branch per: http://djwonk.com/blog/2009/04/18/tracking-remote-git-branches/
          # note: first two lines help avoid  "fatal: Cannot force update the current branch." error per DJWonk.
          git push origin "$branch"
          git checkout master
          git branch -f "$branch" "origin/$branch"
          git checkout "$branch"
        else
          git checkout "$branch"
        fi
      fi
    fi
  else
    echo usage: gbranch \(branch_name\)
  fi
}

Escaping Backticks within Backticks

2010-03-12T17:02:00.007-05:00

If you have trouble with multiple backticks at command-line, it is because you aren't escaping them enough. It takes a lot more to escape then you might think at first. For example, in my last post I provided this example:


find `dirname \`dirname \\\`readlink -f \\\\\\\`which hg\\\\\\\`\\\`\`` -name mercurial

Here is another example:


dirname `dirname \`dirname \\\`dirname \\\\\\\`readlink -f \\\\\\\\\\\\\\\`which hg\\\\\\\\\\\\\\\`\\\\\\\`\\\`\``

So the pattern is that to escape you use 1, 3, 7, 15, 31, etc. where the next number of backticks required is the previous number doubled, plus 1. This can get out-of-hand quickly, so try not to escape backticks within backticks too much if you don't have to.

In addition, the level of backticks affects the number of backslashes required for escaping backslashes (example: Using sed to Escape Quotes).

Converting a Mercurial Repo to Git

2010-03-12T16:55:00.016-05:00

Using OS X 10.6.2 with Mercurial installed (via Macports) and trying to use hg-fast-export.sh that Ben Hood posted about and Paul D. referred me to, I got the error:


ImportError: No module named mercurial

and according to dynamicproxy this meant my PYTHONPATH env variable needed to be setup to include site-packages and site-packages/mercurial dirs.

However, I had trouble at first finding those. In /usr/lib/python2.6/ there was no site-packages. In /Library/Python/2.6/site-packages/ there was no mercurial directory. I had also read that site-packages could be under ~/.local/, but I didn't have a ~/.local/. I figured out that I did in fact have a site-packages dir with mercurial dir in it by doing this:


$ which hg
/opt/local/bin/hg

I found the symlinked dir via:


$ ls -la /opt/local/bin/hg
lrwxr-xr-x  1 root  wheel  66 Nov 30 11:27 /opt/local/bin/hg -> /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin/hg

Then I looked under that dir to find mercurial:


$ find /opt/local/Library/Frameworks/Python.framework/Versions/2.6 -name mercurial
/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial

Of course, with the power of multiple backticks, you could shorten that to:


$ find `dirname \`dirname \\\`readlink -f \\\\\\\`which hg\\\\\\\`\\\`\`` -name mercurial
/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial

I added the following to ~/bash_profile.sh (notice the two paths in PYTHONPATH that were set to the path above and its parent dir) and killed/restarted Terminal.app:


export LC_CTYPE=en_US.UTF-8
export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8
export PYTHONPATH=/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages:/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial:$PYTHONPATH

(note: before blinding copying and pasting that, you need to be sure that is the path to those dirs. See above.)

If you want the repository to be hosted remotely, you may want to create an empty remote git repository that you will push the migrated hg repo to. This step is optional if you are just going to keep the git repo locally.


$ ssh username@someserver
$ sudo su - someotherusername
$ mkdir -p /path/to/remote/repo
$ chmod 775 /path/to/remote/repo
$ cd /path/to/remote/repo
$ git --bare init (with whatever options you want, for example: --shared=group)

Then, Ctrl-D twice to get back to your local environment.

Back on my mac, I was able to successfully use hg-fast-export.sh per Ben Hood here:


$ git clone git://repo.or.cz/fast-export.git
$ mkdir TARGET_REPO
$ cd TARGET_REPO
$ git init-db
$ ../fast-export/hg-fast-export.sh -r PATH_TO_LOCAL_HG_REPO
$ git remote add origin (remote repo path like username@someserver:/path/to/dir )
$ git push origin master

Then just to have a clean clone, go to a directory that should contain your git project directory and do this to get the converted git repo:


$ git clone ssh://username@someserver/path/to/remote/repo

Finally, you will probably want to create at least an additional branch in addition to production for development or testing and then track that branch.

First, per DJWonk's post, you may either want to enable automatic branch tracking globally via:


$ git config --global branch.autosetupmerge true

or do it for this repository:


$ git config branch.autosetupmerge true

Then, do the following to create a local branch (first line), make it remote (second line), and track it (rest of lines) (per Eli Kalderon's post that was recommended in DJWonk's post about tracking git branches):


$ git checkout -b development
$ git push origin development
$ git checkout master
$ git branch -f development origin/development
$ git checkout development

Thunderbird, Mail.app, and Exchange 2007 Rules, Filters, and Calendar Integration

2010-03-11T15:29:00.018-05:00

I've been using Thunderbird in OS X for a few years now. For the most part it has worked pretty well. However, we were just asked to transition accounts from our current mail server to an exchange server. As part of this, we were told that Mail.app and iCal integrate well in Snow Leopard with the version of Exchange that is being used. The first thing I thought was, "How could I migrate/recreate my Thunderbird message filters/rules?" and the second was, "Can't I just use Thunderbird with Lightning for Exchange calendar integration?".

I found that Thunderbird v2 stores its rules in a few places under your Thunderbird profile in OS X: "/Users/username/Library/Thunderbird/Profiles/yourprofileid.default/ImapMail/imap.acme.com/msgFilterRules.dat" and "/Users/username/Library/Thunderbird/Profiles/yourprofileid.default/Mail/Local Folders/msgFilterRules.dat" in a format like:


version="8"
logging="no"
name="Acme Reports"
enabled="yes"
type="1"
action="Move to folder"
actionValue="imap://someusername@imap.acme.org/NameOfMoveToFolder"
condition="OR (to or cc,contains,acme-reports) OR (from,contains,someemail@acme.com) OR (subject,contains,Acme Report)"
... (repeat everything starting with name= over and over for each rule)

How about converting Thunderbird's filters to Mail.app format? Mail.app 4.2 stores its rules in "/Users/username/Library/Mail/MessageRules.plist" in a format like (just using the single rule above as example):


<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
        <key>rules</key>
        <array>
                <dict>
                        <key>Active</key>
                        <string>1</string>
                        <key>AllCriteriaMustBeSatisfied</key>
                        <string>NO</string>
                        <key>AutoResponseType</key>
                        <integer>0</integer>
                        <key>CopyToMailbox</key>
                        <string>~/Library/Mail/IMAP-someusername@imap.acme.org/INBOX/NameOfMoveToFolder.imapmbox</string>
                        <key>CopyToMailboxURL</key>
                        <string>imap://someusername@imap.acme.org/INBOX/NameOfMoveToFolder</string>
                        <key>Criteria</key>
                        <array>
                                <dict>
                                        <key>CriterionUniqueId</key>
                                        <string>a5f2ada7-c0bd-4dff-a1bb-469545f1e5d4</string>
                                        <key>Expression</key>
                                        <string>acme-reports</string>
                                        <key>Header</key>
                                        <string>AnyRecipient</string>
                                </dict>
                                <dict>
                                        <key>CriterionUniqueId</key>
                                        <string>53c7189b-6772-42f8-a0ae-53d238d2d5b0</string>
                                        <key>Expression</key>
                                        <string>someemail@acme.com</string>
                                        <key>Header</key>
                                        <string>From</string>
                                </dict>
                                <dict>
                                        <key>CriterionUniqueId</key>
                                        <string>7952f5ae-22a6-445c-9657-35607d924d8e</string>
                                        <key>Expression</key>
                                        <string>Acme Report</string>
                                        <key>Header</key>
                                        <string>Subject</string>
                                </dict>
                        </array>
                        <key>Deletes</key>
                        <string>NO</string>
                        <key>HighlightTextUsingColor</key>
                        <string>NO</string>
                        <key>Mailbox</key>
                        <string>~/Library/Mail/IMAP-someusername@imap.acme.org/INBOX/NameOfMoveToFolder.imapmbox</string>
                        <key>MailboxURL</key>
                        <string>imap://someusername@imap.acme.org/INBOX/NameOfMoveToFolder</string>
                        <key>MarkFlagged</key>
                        <string>NO</string>
                        <key>MarkRead</key>
                        <string>NO</string>
                        <key>NotifyUser</key>
                        <string>NO</string>
                        <key>RuleId</key>
                        <string>95880085-2370-4026-80F9-AEEA7BE9D4B2</string>
                        <key>RuleName</key>
                        <string>Acme Reports</string>
                        <key>ShouldCopyMessage</key>
                        <string>NO</string>
                        <key>ShouldTransferMessage</key>
                        <string>YES</string>
                        <key>TimeStamp</key>
                        <string>195425409</string>
                </dict>
                <dict>
                        <key>Active</key>
                        <string>1</string>
                        <key>AllCriteriaMustBeSatisfied</key>
                        <string>YES</string>
                        <key>AutoResponseType</key>
                        <integer>0</integer>
                        <key>Color</key>
                        <integer>12096856</integer>
                        <key>Criteria</key>
                        <array>
                                <dict>
                                        <key>CriterionUniqueId</key>
                                        <string>BD45A79F-79EB-4C22-AC0F-98D5B0768C8B</string>
                                        <key>Header</key>
                                        <string>SenderIsNotInAddressBook</string>
                                </dict>
                                <dict>
                                        <key>CriterionUniqueId</key>
                                        <string>623B6201-0B02-4842-BA2A-4196435AD888</string>
                                        <key>Header</key>
                                        <string>SenderIsNotInAddressHistory</string>
                                </dict>
                                <dict>
                                        <key>CriterionUniqueId</key>
                                        <string>83B3EE72-F2CB-4EF2-B060-EFA62209D11B</string>
                                        <key>Header</key>
                                        <string>NoRecipientIncludesFullName</string>
                                </dict>
                                <dict>
                                        <key>CriterionUniqueId</key>
                                        <string>B95AD422-A4C2-4831-B8C6-A7837A23A578</string>
                                        <key>Header</key>
                                        <string>IsJunkMail</string>
                                </dict>
                        </array>
                        <key>Deletes</key>
                        <string>NO</string>
                        <key>HighlightTextUsingColor</key>
                        <string>YES</string>
                        <key>IsOneAndOnlyJunkRule</key>
                        <string>YES</string>
                        <key>MarkFlagged</key>
                        <string>NO</string>
                        <key>MarkRead</key>
                        <string>NO</string>
                        <key>NotifyUser</key>
                        <string>NO</string>
                        <key>RuleId</key>
                        <string>21A4628B-AF2A-4447-A38E-FA1FC2C7FA47</string>
                        <key>RuleName</key>
                        <string>Junk</string>
                        <key>ShouldCopyMessage</key>
                        <string>NO</string>
                        <key>ShouldTransferMessage</key>
                        <string>NO</string>
                        <key>TimeStamp</key>
                        <string>195425409</string>
                        <key>Version</key>
                        <string>2</string>
                </dict>
                <dict>
                        <key>Active</key>
                        <string>1</string>
                        <key>AddInvitationAttachmentToICal</key>
                        <string>YES</string>
                        <key>AllCriteriaMustBeSatisfied</key>
                        <string>YES</string>
                        <key>AutoResponseType</key>
                        <integer>0</integer>
                        <key>Criteria</key>
                        <array>
                                <dict>
                                        <key>CriterionUniqueId</key>
                                        <string>423D56D1-6FA3-4581-9E90-94632EE569D9</string>
                                        <key>Header</key>
                                        <string>AnyMessage</string>
                                </dict>
                        </array>
                        <key>Deletes</key>
                        <string>NO</string>
                        <key>HighlightTextUsingColor</key>
                        <string>NO</string>
                        <key>IsOneAndOnlyICalRule</key>
                        <string>YES</string>
                        <key>MarkFlagged</key>
                        <string>NO</string>
                        <key>MarkRead</key>
                        <string>NO</string>
                        <key>NotifyUser</key>
                        <string>NO</string>
                        <key>RuleId</key>
                        <string>28BF8471-E57D-4685-A059-3855AE1682FD</string>
                        <key>RuleName</key>
                        <string>Add Invitations Automatically</string>
                        <key>ShouldCopyMessage</key>
                        <string>NO</string>
                        <key>ShouldTransferMessage</key>
                        <string>NO</string>
                        <key>TimeStamp</key>
                        <string>195425409</string>
                        <key>Version</key>
                        <string>2</string>
                </dict>
        </array>
        <key>version</key>
        <integer>12</integer>
</dict>
</plist>

So if you wanted to convert Thunderbird message filters to Mail.app rule format, it might take some work to also set the timestamp, UUIDs, etc. inside each rule.

I looked into the Thunderbird Message Filter Import/Export 1.3.7.5 Thunderbird plugin, but it basically exports in the same format as noted above except with two additional lines at the top for its import use:


RootFolderUri=imap://someusername@imap.acme.org
mailnews.customHeaders=

In addition, when Thunderbird starts, that plugin pops up the homepage of the add-on in a browser. I uninstalled it. It seemed just as easy to look at the files I mentioned if you are interested in the rules data, but let's hope that support for exporting filters to Mail.app rules format will be in a future version.

As an alternative to using Mail.app to manage the rules, I was told that Exchange 2007's (rich/IE/"regular") OWA client could be used to setup server-side rules. (Note: Exchange Server 2007's Outlook Web Access Light client must be used if you don't use IE, and you can't manage server-side rules with non-IE browsers in that version of Exchange.) Running Windows XP in VMWare Fusion, and using ieHTTPHeaders to view HTTP Headers in IE, it seemed that the process of adding rules via OWA (non-light/IE version) with Exchange 2007 was a session-based, multi-step process that had several IDs in the XML requests sent to Exchange during the process. It didn't seem very trivial to automate creation of even a single rule via scraping. However, I found reference of an exchange web service here. It looks like the URL to it typically looks like: http://someexchangeserver.somedomain.org/EWS/Exchange.asmx (see MS Exchange 2007 Exchange Web Services Managed API 1.0 docs intro and API docs). It would be nice if there were a plugin in Thunderbird to manage server-side rules in Exchange 2007 server, but I don't see any "Rule" object in the list of data classes.

In addition, while Mail.app was more powerful than Thunderbird 2.0.0.x as far as rules went (for example- you could have it bounce in the dock if gets a high priority mail, or speak a name if an incoming mail matched the rules, etc.), I could not see how to get OWA to allow me to create a rule like "if to or cc contains ____" (you have to specify the exact text, it seemed, rather than to do a "contains").

If you want to migrate rules from Thunderbird 2.0.0.26 to Mail.app v4.2 or Exchange Server 2007, just adding rules by hand in Mail.app or IE OWA client might be easiest, but note that Exchange 2007 OWA's rules may not let you do everything exactly the way you could in Thunderbird. However, if you use server-side rules, then when you connect via the web-based client, rules can be applied. So each have their benefits.

Note that I'm having to switch from one mail server to another for incoming mail, though, so for me, a simple conversion of filters to rules wouldn't work. And because I don't want to have to start VMWare whenever I want to add rules, the OWA Exchange 2007 client is probably not a good option (once they switch to Exchange 2010 though, I might think about using Safari to manage rules, since that is claimed to be supported). I could just manually add folders and rules in Mail.app and that is probably what I'll do.

I am not excluding sticking with Thunderbird either. Maybe Thunderbird's current (or future) integration with Exchange is (or will be) good enough. I installed Thunderbird 3 and Lightning. I'd also like to find Jamon Terrell's add-on as mentioned here. But I haven't found out yet how to hook Lightning/Thunderbird up to Exchange 2007 so that I could see my calendar, others' calendars, and respond to meeting requests easily.

(Specific versions of things I was working with: OS X 10.6.2, Thunderbird v2.0.0.24, Mail(.app) v4.2, and Exchange Server 2007.)

ActiveScaffold and Column Order

2010-03-02T10:02:00.004-05:00

At its simplest column order can be defined in the current version of ActiveScaffold as of 2010-03-02 with just:


active_scaffold :some_model do |config|
  config.columns = [:my_first_column_name, :my_second_column_name, :my_third_column_name, :my_fourth_column_name]
  ...
end

You can also set column weights to specify order like:


config.columns[my_first_column_name].weight = 20

But, if after definition (perhaps in a before_filter), you would like to exclude or add columns, like:


active_scaffold_config.list.columns.exclude :my_first_column_name

and then later do:


active_scaffold_config.list.columns.add :my_first_column_name

Then if you have "config.cache_classes = false" in your config/environments/development.rb, test.rb, and/or production.rb, you will need to have defined column weights for it to display :my_first_column_name as the first column rather than the last.

However, if "config.cache_classes = true" config/environments/development.rb, test.rb, and/or production.rb, and you exclude and add that column back, it will disregard the weight and lose the column order (it will display :my_first_column_name as the last column rather than the first).

A workaround for this provided by Sergio Cambra in issue 732 that worked for us is to specify:


active_scaffold_config.list.columns = active_scaffold_config.columns._inheritable

If you do this, you only need to define config.columns once like:


active_scaffold :some_model do |config|
  config.columns = [:my_first_column_name, :my_second_column_name, :my_third_column_name, :my_fourth_column_name]
  ...
end

and you don't have to define weights. Basically it is resetting the columns to the original set defined. This also means that you don't need to add columns back or exclude columns you added that you want to remove. This worked for us with config.cache_classes set to true or false in development.rb/test.rb/production.rb.

However, if you need ActiveScaffold to keep track of multiple column excludes and adds historically through the session and you are using "config.cache_classes = true", you'd need to modify this workaround. If that is the case, and it is an option to use "config.cache_classes = false", you may just want to define weights and then add and exclude as much as you want.

HotCocoa/MacRuby Links

2010-03-01T11:47:00.006-05:00

Here are some links to help you get started with HotCocoa development as of 2010/03/01:

MacRuby Intro

http://www.macruby.org/documentation/overview.html

HotCocoa Tutorials

HotCocoa Examples

MacRuby/HotCocoa Source

Support

HotCocoa issues: http://github.com/richkilmer/hotcocoa/issues
The only MacRuby list I think that people use: http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
Rich is the HotCocoa guy: http://richkilmer.blogs.com/
Laurent is the MacRuby guy at Apple: http://chopine.be/

Update 2010/06/10

Dan Sinclair (see his fork of HotCocoa) provided some links and an update via an email on the MacRuby-devel mailing list. Here is what he said:

I've got a few tutorials and github apps that I've written with HotCocoa. I kinda stopped poking at it as I wasn't sure what the future held and decided to switch to straight up MacRuby/Cocoa for my last app.
Tutorials
* http://everburning.com/news/heating-up-with-hotcocoa-part-i/
* http://everburning.com/news/heating-up-with-hotcocoa-part-ii/
* http://everburning.com/news/heating-up-with-hotcocoa-part-iii/
* http://everburning.com/news/heating-up-with-hotcocoa-on-github/
* http://everburning.com/news/download-and-xml-parsing-with-hotcocoa/
* http://everburning.com/news/toolbars-with-hotcocoa/
* http://everburning.com/news/hotcocoa-and-core-data/

Github
* http://github.com/dj2/SilverLining (Amazon EC2 Instance Browser)
* http://github.com/dj2/Rife (Start of a news reader)
* http://github.com/dj2/Postie (The app from the heating up tutorials)

dan

Log to Console in Rails Console

2010-02-26T12:53:00.002-05:00

When you are in the Rails console (./script/console), you might want to see what is being logged through the Rails logger in standard out. To do that, just use:


ActiveRecord::Base.logger = Logger.new(STDOUT)

Seconds to Days, Minutes, Hours, Seconds in Ruby

2010-02-26T10:49:00.007-05:00

Here is how to convert a large number of seconds into days, months, hours, seconds format using Ruby (using similar in the Hourz application). total_seconds is the difference between two Ruby Time instances:


  def display_time(total_seconds)
    total_seconds = total_seconds.to_i
    
    days = total_seconds / 86400
    hours = (total_seconds / 3600) - (days * 24)
    minutes = (total_seconds / 60) - (hours * 60) - (days * 1440)
    seconds = total_seconds % 60
    
    display = ''
    display_concat = ''
    if days > 0
      display = display + display_concat + "#{days}d"
      display_concat = ' '
    end
    if hours > 0 || display.length > 0
      display = display + display_concat + "#{hours}h"
      display_concat = ' '
    end
    if minutes > 0 || display.length > 0
      display = display + display_concat + "#{minutes}m"
      display_concat = ' '
    end
    display = display + display_concat + "#{seconds}s"
    display
  end

Example in irb:


>> display_time(Time.at(12345678) - Time.at(1234567))
=> "128d 14h 25m 11s"
>> display_time(Time.at(123456789) - Time.at(12345678))
=> "1286d 0h 11m 51s"

Here is a modification that doesn't show minutes if there are days left and doesn't show seconds if there are days or hours left:


  def display_brief_time(total_seconds)
    total_seconds = total_seconds.to_i
    
    days = total_seconds / 86400
    hours = (total_seconds / 3600) - (days * 24)
    minutes = (total_seconds / 60) - (hours * 60) - (days * 1440)
    seconds = total_seconds % 60
    
    display = ''
    display_concat = ''
    if days > 0
      display = display + display_concat + "#{days}d"
      display_concat = ' '
    end
    if hours > 0 || days > 0
      display = display + display_concat + "#{hours}h"
      display_concat = ' '
    end
    if days == 0 && (minutes > 0 || hours > 0)
      display = display + display_concat + "#{minutes}m"
      display_concat = ' '
    end
    if hours == 0 && days == 0
      display = display + display_concat + "#{seconds}s"
    end
    display
  end

Example of that in irb:


>> display_brief_time(Time.at(12345678) - Time.at(1234567))
=> "128d 14h"
>> display_brief_time(Time.at(12345678) - Time.at(12345677))
=> "1s"
>> display_brief_time(Time.at(12345678) - Time.at(12345607))
=> "1m 11s"
>> display_brief_time(Time.at(12345678) - Time.at(12340607))
=> "1h 24m"

To make it even more brief, you could not show seconds unless there are less than 3 minutes left with:


def display_very_brief_time(total_seconds)
    total_seconds = total_seconds.to_i
    
    days = total_seconds / 86400
    hours = (total_seconds / 3600) - (days * 24)
    minutes = (total_seconds / 60) - (hours * 60) - (days * 1440)
    seconds = total_seconds % 60
    
    display = ''
    display_concat = ''
    if days > 0
      display = display + display_concat + "#{days}d"
      display_concat = ' '
    end
    if hours > 0 || days > 0
      display = display + display_concat + "#{hours}h"
      display_concat = ' '
    end
    if days == 0 && (minutes > 0 || hours > 0)
      display = display + display_concat + "#{minutes}m"
      display_concat = ' '
    end
    if hours == 0 && days == 0 && minutes < 3
      display = display + display_concat + "#{seconds}s"
    end
    display
  end

How to Fix Audio Issues After OS X 10.6 Upgrade from OS X 10.5

2010-02-25T10:16:00.006-05:00

Note: Some mention in this post that this didn't work for them, or the directory wasn't even there, but the fix that Matt Notley noted worked for me! Thanks to Matt and to whoever figured this out, if it wasn't Matt originally!

After being unable to hearing streaming audio via Adobe Acrobat Connect after the upgrade from OS X 10.5 to 10.6 (Snow Leopard) (and subsequent patches to OS X 10.6.2), even after going into System Preferences -> Sound and adjusting volume and setting the device (I use a Eutectics IPP200), I was unable to hear any sound. If I set my sound output to speaker I could hear it fine, but I just wasn't able to play sound for that particular application through that USB device.

Fix

I did a Google search for "snow leopard sound issue" and the top post was "Snow Leopard Sound Bug? - MacTalk Forums". In that post, Matt Notley (mattnotley2004) of Melbourne, VIC states, 'In " Macintosh HD / Library / Preferences / Audio " there are two audio setting files. Delete them. Set your sound to the desired setting. Restart. You may have trouble logging on after a restart, if this happens... just power off your Mac for 10 seconds, then turn it back on. Your sound issue should now be fixed!!'

I backed up the files just in case by just moving them to Desktop like:


sudo mv /Library/Preferences/Audio/* ~/Desktop/

and then closed all applications and restarted. After I rebooted, I opened System Preferences -> Sound and noticed that it retained the setting to use my USB device for sound output.

I also checked to make sure that the /Library/Preferences/Audio/com.apple.audio.DeviceSettings.plist and /Library/Preferences/Audio/com.apple.audio.SystemSettings.plist had been created. I then went back to that same presentation I had previously been unable to hear, and was able to listen to it via my USB device!

What Did This Change?

For those wishing to see what the difference was in the files- I can't say for certain that the ownership of the files changed, but the new owner is _coreaudiod. Here are the diffs, showing that the "input streams" key and settings section was missing from com.apple.audio.DeviceSettings.plist. This is a diff of old (broken) vs. new (working) files:


$ diff /Users/.../Desktop/com.apple.audio.DeviceSettings.plist /Library/Preferences/Audio/com.apple.audio.DeviceSettings.plist
434c434
<     <real>1</real>
---
>     <real>0.77777779102325439</real>
444c444
<     <real>1</real>
---
>     <real>0.77777779102325439</real>
700a701,1066
>   <key>input streams</key>
>   <array>
>    <dict>
>     <key>bits per channel</key>
>     <integer>32</integer>
>     <key>bytes per frame</key>
>     <integer>4</integer>
>     <key>bytes per packet</key>
>     <integer>4</integer>
>     <key>channels per frame</key>
>     <integer>1</integer>
>     <key>format flags</key>
>     <integer>25</integer>
>     <key>format id</key>
>     <integer>1819304813</integer>
>     <key>frames per packet</key>
>     <integer>1</integer>
>     <key>sample rate</key>
>     <real>44100</real>
>    </dict>
>    <dict>
>     <key>bits per channel</key>
>     <integer>32</integer>
>     <key>bytes per frame</key>
>     <integer>4</integer>
>     <key>bytes per packet</key>
>     <integer>4</integer>
>     <key>channels per frame</key>
>     <integer>1</integer>
>     <key>format flags</key>
>     <integer>25</integer>
>     <key>format id</key>
>     <integer>1819304813</integer>
>     <key>frames per packet</key>
>     <integer>1</integer>
>     <key>sample rate</key>
>     <real>44100</real>
>    </dict>
>    <dict>
>     <key>bits per channel</key>
>     <integer>32</integer>
>     <key>bytes per frame</key>
>     <integer>4</integer>
>     <key>bytes per packet</key>
>     <integer>4</integer>
>     <key>channels per frame</key>
>     <integer>1</integer>
>     <key>format flags</key>
>     <integer>25</integer>
>     <key>format id</key>
>     <integer>1819304813</integer>
>     <key>frames per packet</key>
>     <integer>1</integer>
>     <key>sample rate</key>
>     <real>44100</real>
>    </dict>
>    <dict>
>     <key>bits per channel</key>
>     <integer>32</integer>
>     <key>bytes per frame</key>
>     <integer>4</integer>
>     <key>bytes per packet</key>
>     <integer>4</integer>
>     <key>channels per frame</key>
>     <integer>1</integer>
>     <key>format flags</key>
>     <integer>25</integer>
>     <key>format id</key>
>     <integer>1819304813</integer>
>     <key>frames per packet</key>
>     <integer>1</integer>
>     <key>sample rate</key>
>     <real>44100</real>
>    </dict>
>    <dict>
>     <key>bits per channel</key>
>     <integer>32</integer>
>     <key>bytes per frame</key>
>     <integer>4</integer>
>     <key>bytes per packet</key>
>     <integer>4</integer>
>     <key>channels per frame</key>
>     <integer>1</integer>
>     <key>format flags</key>
>     <integer>25</integer>
>     <key>format id</key>
>     <integer>1819304813</integer>
>     <key>frames per packet</key>
>     <integer>1</integer>
>     <key>sample rate</key>
>     <real>44100</real>
>    </dict>
>    <dict>
>     <key>bits per channel</key>
>     <integer>32</integer>
>     <key>bytes per frame</key>
>     <integer>4</integer>
>     <key>bytes per packet</key>
>     <integer>4</integer>
>     <key>channels per frame</key>
>     <integer>1</integer>
>     <key>format flags</key>
>     <integer>25</integer>
>     <key>format id</key>
>     <integer>1819304813</integer>
>     <key>frames per packet</key>
>     <integer>1</integer>
>     <key>sample rate</key>
>     <real>44100</real>
>    </dict>
>    <dict>
>     <key>bits per channel</key>
>     <integer>32</integer>
>     <key>bytes per frame</key>
>     <integer>4</integer>
>     <key>bytes per packet</key>
>     <integer>4</integer>
>     <key>channels per frame</key>
>     <integer>1</integer>
>     <key>format flags</key>
>     <integer>25</integer>
>     <key>format id</key>
>     <integer>1819304813</integer>
>     <key>frames per packet</key>
>     <integer>1</integer>
>     <key>sample rate</key>
>     <real>44100</real>
>    </dict>
>    <dict>
>     <key>bits per channel</key>
>     <integer>32</integer>
>     <key>bytes per frame</key>
>     <integer>4</integer>
>     <key>bytes per packet</key>
>     <integer>4</integer>
>     <key>channels per frame</key>
>     <integer>1</integer>
>     <key>format flags</key>
>     <integer>25</integer>
>     <key>format id</key>
>     <integer>1819304813</integer>
>     <key>frames per packet</key>
>     <integer>1</integer>
>     <key>sample rate</key>
>     <real>44100</real>
>    </dict>
>    <dict>
>     <key>bits per channel</key>
>     <integer>32</integer>
>     <key>bytes per frame</key>
>     <integer>4</integer>
>     <key>bytes per packet</key>
>     <integer>4</integer>
>     <key>channels per frame</key>
>     <integer>1</integer>
>     <key>format flags</key>
>     <integer>25</integer>
>     <key>format id</key>
>     <integer>1819304813</integer>
>     <key>frames per packet</key>
>     <integer>1</integer>
>     <key>sample rate</key>
>     <real>44100</real>
>    </dict>
>    <dict>
>     <key>bits per channel</key>
>     <integer>32</integer>
>     <key>bytes per frame</key>
>     <integer>4</integer>
>     <key>bytes per packet</key>
>     <integer>4</integer>
>     <key>channels per frame</key>
>     <integer>1</integer>
>     <key>format flags</key>
>     <integer>25</integer>
>     <key>format id</key>
>     <integer>1819304813</integer>
>     <key>frames per packet</key>
>     <integer>1</integer>
>     <key>sample rate</key>
>     <real>44100</real>
>    </dict>
>   </array>
>   <key>output streams</key>
>   <array>
>    <dict>
>     <key>bits per channel</key>
>     <integer>32</integer>
>     <key>bytes per frame</key>
>     <integer>4</integer>
>     <key>bytes per packet</key>
>     <integer>4</integer>
>     <key>channels per frame</key>
>     <integer>1</integer>
>     <key>format flags</key>
>     <integer>25</integer>
>     <key>format id</key>
>     <integer>1819304813</integer>
>     <key>frames per packet</key>
>     <integer>1</integer>
>     <key>sample rate</key>
>     <real>44100</real>
>    </dict>
>    <dict>
>     <key>bits per channel</key>
>     <integer>32</integer>
>     <key>bytes per frame</key>
>     <integer>4</integer>
>     <key>bytes per packet</key>
>     <integer>4</integer>
>     <key>channels per frame</key>
>     <integer>1</integer>
>     <key>format flags</key>
>     <integer>25</integer>
>     <key>format id</key>
>     <integer>1819304813</integer>
>     <key>frames per packet</key>
>     <integer>1</integer>
>     <key>sample rate</key>
>     <real>44100</real>
>    </dict>
>    <dict>
>     <key>bits per channel</key>
>     <integer>32</integer>
>     <key>bytes per frame</key>
>     <integer>4</integer>
>     <key>bytes per packet</key>
>     <integer>4</integer>
>     <key>channels per frame</key>
>     <integer>1</integer>
>     <key>format flags</key>
>     <integer>25</integer>
>     <key>format id</key>
>     <integer>1819304813</integer>
>     <key>frames per packet</key>
>     <integer>1</integer>
>     <key>sample rate</key>
>     <real>44100</real>
>    </dict>
>    <dict>
>     <key>bits per channel</key>
>     <integer>32</integer>
>     <key>bytes per frame</key>
>     <integer>4</integer>
>     <key>bytes per packet</key>
>     <integer>4</integer>
>     <key>channels per frame</key>
>     <integer>1</integer>
>     <key>format flags</key>
>     <integer>25</integer>
>     <key>format id</key>
>     <integer>1819304813</integer>
>     <key>frames per packet</key>
>     <integer>1</integer>
>     <key>sample rate</key>
>     <real>44100</real>
>    </dict>
>    <dict>
>     <key>bits per channel</key>
>     <integer>32</integer>
>     <key>bytes per frame</key>
>     <integer>4</integer>
>     <key>bytes per packet</key>
>     <integer>4</integer>
>     <key>channels per frame</key>
>     <integer>1</integer>
>     <key>format flags</key>
>     <integer>25</integer>
>     <key>format id</key>
>     <integer>1819304813</integer>
>     <key>frames per packet</key>
>     <integer>1</integer>
>     <key>sample rate</key>
>     <real>44100</real>
>    </dict>
>    <dict>
>     <key>bits per channel</key>
>     <integer>32</integer>
>     <key>bytes per frame</key>
>     <integer>4</integer>
>     <key>bytes per packet</key>
>     <integer>4</integer>
>     <key>channels per frame</key>
>     <integer>1</integer>
>     <key>format flags</key>
>     <integer>25</integer>
>     <key>format id</key>
>     <integer>1819304813</integer>
>     <key>frames per packet</key>
>     <integer>1</integer>
>     <key>sample rate</key>
>     <real>44100</real>
>    </dict>
>    <dict>
>     <key>bits per channel</key>
>     <integer>32</integer>
>     <key>bytes per frame</key>
>     <integer>4</integer>
>     <key>bytes per packet</key>
>     <integer>4</integer>
>     <key>channels per frame</key>
>     <integer>1</integer>
>     <key>format flags</key>
>     <integer>25</integer>
>     <key>format id</key>
>     <integer>1819304813</integer>
>     <key>frames per packet</key>
>     <integer>1</integer>
>     <key>sample rate</key>
>     <real>44100</real>
>    </dict>
>    <dict>
>     <key>bits per channel</key>
>     <integer>32</integer>
>     <key>bytes per frame</key>
>     <integer>4</integer>
>     <key>bytes per packet</key>
>     <integer>4</integer>
>     <key>channels per frame</key>
>     <integer>1</integer>
>     <key>format flags</key>
>     <integer>25</integer>
>     <key>format id</key>
>     <integer>1819304813</integer>
>     <key>frames per packet</key>
>     <integer>1</integer>
>     <key>sample rate</key>
>     <real>44100</real>
>    </dict>
>    <dict>
>     <key>bits per channel</key>
>     <integer>32</integer>
>     <key>bytes per frame</key>
>     <integer>4</integer>
>     <key>bytes per packet</key>
>     <integer>4</integer>
>     <key>channels per frame</key>
>     <integer>1</integer>
>     <key>format flags</key>
>     <integer>25</integer>
>     <key>format id</key>
>     <integer>1819304813</integer>
>     <key>frames per packet</key>
>     <integer>1</integer>
>     <key>sample rate</key>
>     <real>44100</real>
>    </dict>
>    <dict>
>     <key>bits per channel</key>
>     <integer>32</integer>
>     <key>bytes per frame</key>
>     <integer>4</integer>
>     <key>bytes per packet</key>
>     <integer>4</integer>
>     <key>channels per frame</key>
>     <integer>1</integer>
>     <key>format flags</key>
>     <integer>25</integer>
>     <key>format id</key>
>     <integer>1819304813</integer>
>     <key>frames per packet</key>
>     <integer>1</integer>
>     <key>sample rate</key>
>     <real>44100</real>
>    </dict>
>   </array>

Here is the diff of com.apple.audio.SystemSettings.plist (old broken vs. new fixed). It made some changes and removed several keys:


$ diff /Users/.../Desktop/com.apple.audio.SystemSettings.plist /Library/Preferences/Audio/com.apple.audio.SystemSettings.plist
8c8
<   <string>AppleHDAEngineInput:1B,0,1,0:3</string>
---
>   <string>AppleUSBAudioEngine:Eutectics Inc.:Eutectics IPP200:07162001-01:2</string>
10,12c10
<   <string>AppleHDAEngineOutput:1B,0,1,2:0</string>
<   <key>current default system output device</key>
<   <string>AppleHDAEngineOutput:1B,0,1,2:0</string>
---
>   <string>AppleUSBAudioEngine:Eutectics Inc.:Eutectics IPP200:07162001-01:1</string>
22,36d19
<   <key>primary default input device</key>
<   <string>AppleUSBAudioEngine:Eutectics Inc.:Eutectics IPP200:07162001-01:2</string>
<   <key>primary default output device</key>
<   <string>AppleUSBAudioEngine:Eutectics Inc.:Eutectics IPP200:07162001-01:1</string>
<   <key>primary default system output device</key>
<   <string>AppleUSBAudioEngine:Eutectics Inc.:Eutectics IPP200:07162001-01:1</string>
<   <key>secondary default output device</key>
<   <string>AppleHDAEngineOutput:1B,0,1,2:0</string>
<   <key>secondary default system output device</key>
<   <string>AppleHDAEngineOutput:1B,0,1,2:0</string>
<  </dict>
<  <key>92</key>
<  <dict>
<   <key>current default system output device</key>
<   <string>AppleHDAEngineOutput:1B,0,1,2:0</string>

hg serve and hg glog

2010-02-23T15:55:00.016-05:00

If you want to view Mercurial changes graphically, there are at least two choices: hg serve (web-based version) and hg glog (command-line version).

For the first, you do:


hg serve

then go to http://localhost:8000/, then click "graph" on the top left, and by repeated clicking "more" on the top right, you will find the changes you want to see.

We pushed the graph of changes to the limit today and we couldn't fully see the stuff we were interested on the far right, because it was cut-off:

Richard told me about the Graph log extension for Mercurial to see branches at command-line (in Terminal.app), which was able to do a better job at displaying the change history graphically than hg serve did.

To use that, edit ~/.hgrc and add "hgext.graphlog = " under [extensions]. Then:


hg glog

which produces output like the following that doesn't break. I attempted to use the same changesets as shown above in the pic for an apples-to-apples comparison, even though it looks a bit different:


| o---------+ | |  changeset:   ...
|   | | | | | | |  user:        ...
|  / / / / / / /   date:        ...
| | | | | | | |    summary:     ...
| | | | | | | |
| | | | | o | |    changeset:   ...
| | | | | |\ \ \   parent:      ...
| | | | | | | | |  parent:      ...
| | | | | | | | |  user:        ...
| | | | | | | | |  date:        ...
| | | | | | | | |  summary:     ...
| | | | | | | | |
| | +-------o | |  changeset:   ...
| | | | | | | | |  parent:      ...
| | | | | | | | |  parent:      ...
| | | | | | | | |  user:        ...
| | | | | | | | |  date:        ...
| | | | | | | | |  summary:     ...
| | | | | | | | |
o---------+ | | |  changeset:   ...
| | | | | | | | |  parent:      ...
| | | | | | | | |  parent:      ...
| | | | | | | | |  user:        ...
| | | | | | | | |  date:        ...
| | | | | | | | |  summary:     ...
| | | | | | | | |
| | | | | o | | |  changeset:   ...
| | | | | | | | |  parent:      ...
| | | | | | | | |  user:        ...
| | | | | | | | |  date:        ...
| | | | | | | | |  summary:     ...
| | | | | | | | |
o-----------+ | |  changeset:   ...
| | | | | | | | |  parent:      ...
| | | | | | | | |  parent:      ...
| | | | | | | | |  user:        ...
| | | | | | | | |  date:        ...
| | | | | | | | |  summary:     ...
| | | | | | | | |
| | | | | | o | |  changeset:   ...
| | | | | | | | |  parent:      ...
| | | | | | | | |  user:        ...
| | | | | | | | |  date:        ...
| | | | | | | | |  summary:     ...
| | | | | | | | |
o-------------+ |  changeset:   ...
| | | | | | | | |  parent:      ...
| | | | | | | | |  parent:      ...
| | | | | | | | |  user:        ...
| | | | | | | | |  date:        ...
| | | | | | | | |  summary:     ...
| | | | | | | | |
| | | | | | | o |  changeset:   ...
| | | | | | | |/   parent:      ...
| | | | | | | |    user:        ...
| | | | | | | |    date:        ...
| | | | | | | |    summary:     ...
| | | | | | | |
o-------------+  changeset:   ...
| | | | | | | |  parent:      ...
| | | | | | | |  parent:      ...
| | | | | | | |  user:        ...
| | | | | | | |  date:        ...
| | | | | | | |  summary:     ...
| | | | | | | |
| | +---------o  changeset:   ...
| | | | | | |    parent:      ...
| | | | | | |    user:        ...
| | | | | | |    date:        ...
| | | | | | |    summary:     ...
| | | | | | |
o-----------+  changeset:   ...
| | | | | | |  parent:      ...
| | | | | | |  parent:      ...
| | | | | | |  user:        ...
| | | | | | |  date:        ...
| | | | | | |  summary:     ...
| | | | | | |
| | | | | | o  changeset:   ...
| | | | | | |  parent:      ...
| | | | | | |  user:        ...
| | | | | | |  date:        ...
| | | | | | |  summary:     ...
| | | | | | |
o | | | | | |  changeset:   ...
|\| | | | | |  parent:      ...
| | | | | | |  parent:      ...
| | | | | | |  user:        ...
| | | | | | |  date:        ...
| | | | | | |  summary:     ...
| | | | | | |
| o | | | | |  changeset:   ...
| | | | | | |  parent:      ...
| | | | | | |  user:        ...
| | | | | | |  date:        ...
| | | | | | |  summary:     ...
| | | | | | |
o | | | | | |  changeset:   ...
|\| | | | | |  parent:      ...
| | | | | | |  parent:      ...
| | | | | | |  user:        ...
| | | | | | |  date:        ...
| | | | | | |  summary:     ...
| | | | | | |
| o | | | | |  changeset:   ...
| | | | | | |  parent:      ...
| | | | | | |  user:        ...
| | | | | | |  date:        ...
| | | | | | |  summary:     ...
| | | | | | |
o | | | | | |  changeset:   ...
|\| | | | | |  parent:      ...
| | | | | | |  parent:      ...
| | | | | | |  user:        ...
| | | | | | |  date:        ...
| | | | | | |  summary:     ...
| | | | | | |
| o | | | | |  changeset:   ...
| | | | | | |  parent:      ...
| | | | | | |  user:        ...
| | | | | | |  date:        ...
| | | | | | |  summary:     ...
| | | | | | |
o-----------+  changeset:   ...
| | | | | | |  parent:      ...
| | | | | | |  parent:      ...
| | | | | | |  user:        ...
| | | | | | |  date:        ...
| | | | | | |  summary:     ...

Read a File to a String in Ruby

2010-02-19T10:46:00.010-05:00

Reading a file into a String in Ruby is much easier than most examples I've found show that it is, probably because they are old. Reading a File into a String in Ruby is just:


  f = File.open('filename','r')
  s = f.read

Looking at ActiveRecord, I noticed that the implementations of from_json used something similar to the following code in the beginning of each method:


  def output(some_input)
    if respond_to? :read
      some_input = some_input.read
    end
    p some_input
  end

So by including those three lines, you add the ability for any method to either take a String or a File as the parameter/argument, so your method can a File:


  f = File.open('filename','r')
  output(f)

just as easily as it can take a String:


  output('Hello, World!')

HotCocoa App to Track Time on Tasks

2010-02-17T16:27:00.002-05:00

Just wrote a HotCocoa/MacRuby OS X app called Hourz to track time on tasks. HotCocoa and MacRuby are both pretty awesome, and the Cocoa framework obviously is even more-so. I was able to get a quick application up and running quickly and of course make a neat little icon to show up in the dock.

That said, I did run into some issues. However, it is what you expect when learning a new framework and a new language that are both relatively new also. However, I expect if there is continued effort in both MacRuby and HotCocoa development, it might become a great way to write OS X applications quickly even for someone not familiar with Cocoa.

MacRuby IRB Console

2010-02-16T16:09:00.002-05:00

Just use the following at command-line to start up a MacRuby IRB session:


macirb

Create a Custom Icon for Your HotCocoa App

2010-02-16T10:15:00.012-05:00

If you've written an application for Mac OS X in HotCocoa, one of the first things you probably want to do is to change the icon. This is simple to do, but I didn't see it documented anywhere.

The following process uses tools freely available for OS X (I use OS X 10.6.2) to create and change the icon.

First, grab Seashore if you need an image editor. If you have a non-rectangular image and want to use a transparency, use the tip here. If you want it to look like the rest of the icons, see Icons in the Apple Human Interface Guidelines document on Apple's site. If you have photoshop you could try something like this. But, unless you need it to look professional, just save the image.

Open Img2icns and drag the image into it. The free version works fine for this, but I recommend buying the pro version, even if only to fund their development. You can tell where the transparency will be in the icon by where you see the blueish background within the area. Then click on Icns in Img2icns to create the icns file.

Copy that file to the resources directory in your HotCocoa project and name it (name of app).icns. If you are not sure what to name it, look in the built application (after calling macrake) that resides in (name of app).app/Contents/Resources/. By default the HotCocoa macrake build copies HotCocoa.icns to (app name).icns unless you provide a (name of app).icns in the (project name)/resources directory.

Call macrake and see the new icon for your app! If you already had your app in the dock, you might need to remove it from the dock to see the icon change.

Call Some Toll-free Numbers Free Internationally Using Skype

2010-02-15T09:56:00.005-05:00

Recently on the uportal-users list Conan Cook was asking how they could call a U.S. Toll-free phone number from the UK, and Cris Holdorph replied that he could use Skype. I just looked into it and Skype has a list of the countries that have toll-free numbers that you can call from anywhere using Skype (using SkypeOut, even though you don't have to pay for it). Just do a search in the page for "Toll Free" to see which countries' toll-free numbers you can call with Skype:
* http://www.skype.com/prices/callrates/#allRatesTab

Get Request Param Values in Javascript

2010-02-11T11:00:00.015-05:00

The following should get any request parameter from the URL, escaping all characters used in regular expressions:


function getReqParam(name)
{
  var regexpstr = "[\\?&]"+escapeRegExp(name)+"=([^&#]*)";
  var arr = new RegExp(regexpstr).exec(window.location.href);
  if(arr == null) return null;
  return arr[1];
}

function escapeRegExp(value) {
  return value.replace(/\\/g,"\\\\").replace(/\//g,"\\\/").replace(/\./g,"\\\.").replace(/\*/g,"\\\*").replace(/\+/g,"\\\+").replace(/\?/g,"\\\?").replace(/\|/g,"\\\|").replace(/\(/g,"\\\(").replace(/\)/g,"\\\)").replace(/\[/g,"\\\[").replace(/\]/g,"\\\]").replace(/\{/g,"\\\{").replace(/\}/g,"\\\}");
}

Here is a example test HTML that you can use to see how it works by specifying the request parameter as the target (like ?someparam=somevalue#someparam), just for the heck of it.


<html>
<head><title>test</title></head>
<body>
<script language="javascript" type="text/javascript" defer>
<!--
function getReqParam(name)
{
  var regexpstr = "[\\?&]"+escapeRegExp(name)+"=([^&#]*)";
  var arr = new RegExp(regexpstr).exec(window.location.href);
  if(arr == null) return null;
  return arr[1];
}

function escapeRegExp(value) {
  return value.replace(/\\/g,"\\\\").replace(/\//g,"\\\/").replace(/\./g,"\\\.").replace(/\*/g,"\\\*").replace(/\+/g,"\\\+").replace(/\?/g,"\\\?").replace(/\|/g,"\\\|").replace(/\(/g,"\\\(").replace(/\)/g,"\\\)").replace(/\[/g,"\\\[").replace(/\]/g,"\\\]").replace(/\{/g,"\\\{").replace(/\}/g,"\\\}");
}

window.location.href="?\/.*+?|(){}[]{}=asdf#\/.*+?|(){}[]{}"
if (window.location.hash == null || window.location.hash.length < 2) {
  document.write("modify the url to specify req param and hash (target) is the req param name")
} else {
  document.write( "request param name '" + window.location.hash + "'=" + getReqParam(window.location.hash.substr(1)) + "<br/>");
}
-->
</script>
</body></html>

I didn't find any other examples that allowed all of the characters ""/.*+?|(){}[]{}"" in the request parameter name, even though the likelihood of that is low for most applications.

Hash.from_xml and Multiple Elements with Same Name

2010-02-05T10:25:00.008-05:00

I've seen reference to ActiveSupport's Hash.from_xml method to easily parse XML into hashes, but I assumed that it would overwrite keys when the same element name was defined multiple times while in the same context, as you would think it might, but it doesn't (which is a good thing).

Some examples:


Hash.from_xml "<a><b><tags><tag>apple</tag><tag>orange</tag></tags></b></a>"

produces the hash:


{"a"=>{"b"=>{"tags"=>{"tag"=>["apple", "orange"]}}}}

and


Hash.from_xml "<a><b><a>1</a><b>apple</b><a>2</a><b>banana</b></b></a>"

produces the hash:


{"a"=>{"b"=>{"a"=>["1", "2"], "b"=>["apple", "banana"]}}}

Please note that well-designed XML should usually not use unencapsulated key/value pairs in a XML like:


<keypairs><key>key1</key><value>value1</value><key>key2</key><value>value2</value></keypairs>

but I've had to consume XML like this before (very recently).

Note that XML attributes can be lost as noted in ticket #1598, but perhaps that will be fixed in a future version of ActiveSupport/Rails.

If you get "NoMethodError: undefined method `from_xml' for Hash:Class", remember that Rails includes ActiveSupport. If you're not using Rails, you need to:


require 'rubygems'                          
require 'active_support'

After playing with REXML and XmlSimple a little previously, this seems to be one of the easier ways to consume XML in Ruby/JRuby, if you want to work with most of the data in the supplied in the XML (and if you either don't need XML attributes or ticket #1598 gets fixed). However, it isn't always that great. Depending on the xml, it might produce hashes and arrays sometimes and not other times when multiples are not defined, ending with code that has respond_to? :each and respond_to? :keys , etc. So, it is not a solution for every case, but it sometimes provides a fairly simple solution.

Create a MacRuby HotCocoa App

2010-02-02T13:50:00.013-05:00

You can create an OS X app quickly in Snow Leopard (OS 10.6.x) or later with MacRuby and HotCocoa.

Much of this information was provided in Isaac Kearse's post, Packaging A HotCocoa Application, which I'd recommend reading also as it has more information than I've included, and a lot of this is copied as a backup from Isaac's post. It's Ruby, so a lot of it should be familiar.

1. Download MacRuby.

2. Double-click on the zip file to extract it and double-click on the package inside the extracted directory to start the install. Just go with the defaults.

3. You may want to kill off terminal.app if you have it running and restart it, just in case the install put anything into the environment.

4. Use the following to install the hotcocoa macgem:


sudo macgem install hotcocoa

5. Change to the directory that you want your project directory to reside in:


mkdir -p /path/to/my/projects/
cd /path/to/my/projects/

6. Create a sample hotcocoa app:


hotcocoa nameofyourapp

Once you've done what you want, you can build and run your sample hotcocoa app via:


cd nameofyourapp
macrake

and deploy it via:


macrake deploy

If it won't work, run it so that you can see the error in terminal.app:


macruby lib/application.rb --debug

If you're curious, you can see what other macgems are available via:


sudo macgem list --remote

If you want to include a custom icon, see my other post.

Again, thanks much to Isaac Kearse's post, Packaging A HotCocoa Application, which contains even more information than I've provided here.

Show Name of Git Branch in Prompt

2010-01-28T10:17:00.012-05:00

The following shows you how to have a nifty prompt in OS X like


[myhostname dirname (git branch name)]$

when you are in a git project, and


[myhostname dirname]$

when you aren't. Works for me in OS X 10.6.2 (Snow Leopard) and for another co-worker that tried it in OS X 10.5.x.

Basically you just need to call the bash script that does git completion, and then set the prompt.

Updated Aug 18, 2010 with instructions for Homebrew.

Using Homebrew

This assumes you installed Git via Homebrew. If you haven't installed Homebrew or Git yet, the instructions are on the Homebrew site, which I'd recommend, but here are the commands to do that, as of August 2010:


ruby -e "$(curl -fsS http://gist.github.com/raw/323731/install_homebrew.rb)"
brew install wget
brew install git
brew update

then add this to the bottom of your ~/.bash_profile, ~/.profile, or ~/.zshrc:


# see: http://stufftohelpyouout.blogspot.com/2010/01/show-name-of-git-branch-in-prompt.html
# see also: http://superuser.com/questions/31744/how-to-get-git-completion-bash-to-work-on-mac-os-x
# see also: http://stackoverflow.com/questions/347901/what-are-your-favorite-git-features-or-tricks
# This assumes you have installed Homebrew ( http://github.com/mxcl/homebrew )
# and then installed Git via Homebrew with the default installation location:
# ruby -e "$(curl -fsS http://gist.github.com/raw/323731/install_homebrew.rb)"
# brew install wget
# brew install git
# brew update
if [ -f /usr/local/etc/bash_completion.d/git-completion.bash ]; then
  . /usr/local/etc/bash_completion.d/git-completion.bash;
  PS1='[\h \W$(__git_ps1 " (%s)")]\$ ';
fi

Using MacPorts

For this I had installed Git via the Google Code git-osx-installer.

First install MacPorts. Then do:


sudo port selfupdate
sudo port install git-core +bash_completion

then add this to the bottom of your ~/.bash_profile, ~/.profile, or ~/.zshrc:


# see: http://stufftohelpyouout.blogspot.com/2010/01/show-name-of-git-branch-in-prompt.html
# see also: http://superuser.com/questions/31744/how-to-get-git-completion-bash-to-work-on-mac-os-x
# see also: http://stackoverflow.com/questions/347901/what-are-your-favorite-git-features-or-tricks
# install http://macports.org/
# sudo port selfupdate
# sudo port install git-core +bash_completion
if [ -f /opt/local/etc/bash_completion ]; then
    . /opt/local/etc/bash_completion
    PS1='[\h \W$(__git_ps1 " (%s)")]\$ '
fi

Then completely close Terminal.app and reopen it. When you cd into a dir containing a git project, you will see the currently checked out git branch within parentheses at the end of the prompt.

This solution is based on this prompt tip from Casey and the setup and related bit needed in .bash_profile is based on this tip on setting up git-completion from Adam Johnson.

2 Minute Guide to Remote Git Repository Creation

2010-01-27T11:31:00.017-05:00

This is basically the same as the 37signals Git Resources Creating a remote repository section, with some minor modifications and clarifications. Both are geared toward someone who wants to create a repository accessible to only people in certain group.

First off, login to the remote server:


ssh userid@yourremoteserver

Let's assume that you already setup a user "otheruserid" (maybe you created a "git" user) that is in the same group as the developers that need access to the repository. We'll assume that "userid" is a member of that same group. It is a better idea to create this as a more generic user that won't be removed later, so that the group membership can be dynamic (as people come and go). Note that the .git at the end of the path is optional.


sudo su - otheruserid
mkdir -p /some/path/nameofmyproject.git
chmod 775 /some/path/nameofmyproject.git
cd /some/path/nameofmyproject.git

For this next part, don't change group to the group name. It is literally supposed to be this exact line if you want to share with the group. If you want it to be open to the public, you'd specify "all" instead of "group".


git --bare init --shared=group
(Ctrl-D twice to get back to your local environment)

Richard mentioned that if you want to create a private group repository and manage permissions across everything a little better you could use getacl/setacl (even though you should still use --shared=group, just in case). You can probably find more on permissions via a search, but berliOS has a GIT HowTo with a section on How to control access to your GIT Repository just as one example of fine-grained permissions.

Back on your computer, or wherever you want to develop, do the following:


mkdir -p /path/to/my/projects/dir
cd /path/to/my/projects/dir
mkdir nameofmyproject

Be careful that you change directory into your new nameofmyproject directory, otherwise you could end up adding everything that is not nailed down (via git) into your new repository!


cd nameofmyproject
git init
git remote add origin userid@yourremoteserver:/some/path/nameofmyproject.git
touch .gitignore
git add .
git commit -m "Initial commit"
git push --all

Ok, now you have the local copy, but if you try to do "git pull" it won't work because the following isn't defined in .git/config:


[branch "master"]
 remote = origin
 merge = refs/heads/master

You could edit .git/config to add that to the bottom, or better yet- blow away the local project directory you just created (please back it up first if it is important!) and clone from the remote repo:


cd ..
rm -rf nameofproject (warning: back it up first, if you are unsure about removing it)
git clone ssh://userid@yourremoteserver/some/path/nameofmyproject.git

If someone else wants to clone the repository to work with it locally, they do this:


git clone ssh://userid@yourremoteserver/some/path/nameofmyproject.git

Remotely Determining Ruby, RubyGems, and Gem Versions via ssh

2010-01-25T12:16:00.008-05:00

The following one liner displays the ruby executable's version, RUBY_VERSION, the RubyGems version, and lists all gems:


ssh userid@server 'ruby -v && echo RUBY_VERSION: && ruby -e "p RUBY_VERSION" && echo RubyGems version: && gem -v && echo Installed gems: && gem list'

Send Capshell Commands in Single Command-line

2010-01-14T14:26:00.007-05:00

You can echo multiple lines and pipe them into capshell like:


echo "some command 1" "some command 2" | capshell

That starts capshell and runs some command 1 and then some command 2. Simple, but I thought it was neat, especially if you just want to create an alias to check for something via a short command line like:


alias e=cd /path/to/rails/project && echo "with appservers" "grep -r some_text /path/on/server/to/app/current/log" | capshell

then at command line you just type 'e' and it will grep all app server logs for some_text.

For more info about capshell, see Capistrano.

Fix for undefined method `define_a_column' for class `OCI8::Cursor' (NameError) in OS X 10.6 (Snow Leopard)

2010-01-05T12:44:00.013-05:00

If you're using Ruby with ruby-oci8 in OS X 10.6 (Snow Leopard), you may get the following error:


$ ./script/server
=> Booting WEBrick...
./script/../config/../vendor/rails/activerecord/lib/active_record/connection_adapters/oracle_adapter.rb:536: undefined method `define_a_column' for class `OCI8::Cursor' (NameError)
from /Library/Ruby/Site/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
from /Library/Ruby/Site/1.8/rubygems/custom_require.rb:31:in `require'
from ./script/../config/../vendor/rails/activesupport/lib/active_support/dependencies.rb:495:in `require'
from ./script/../config/../vendor/rails/activesupport/lib/active_support/dependencies.rb:342:in `new_constants_in'
from ./script/../config/../vendor/rails/activesupport/lib/active_support/dependencies.rb:495:in `require'
from ./script/../config/../vendor/rails/activerecord/lib/active_record.rb:81
from ./script/../config/../vendor/rails/activerecord/lib/active_record.rb:80:in `each'
from ./script/../config/../vendor/rails/activerecord/lib/active_record.rb:80
from /Library/Ruby/Site/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
from /Library/Ruby/Site/1.8/rubygems/custom_require.rb:31:in `require'
from ./script/../config/../vendor/rails/activesupport/lib/active_support/dependencies.rb:495:in `require'
from ./script/../config/../vendor/rails/activesupport/lib/active_support/dependencies.rb:342:in `new_constants_in'
from ./script/../config/../vendor/rails/activesupport/lib/active_support/dependencies.rb:495:in `require'
from ./script/../config/../vendor/rails/railties/lib/initializer.rb:162:in `require_frameworks'
from ./script/../config/../vendor/rails/railties/lib/initializer.rb:162:in `each'
from ./script/../config/../vendor/rails/railties/lib/initializer.rb:162:in `require_frameworks'
from ./script/../config/../vendor/rails/railties/lib/initializer.rb:83:in `process'
from ./script/../config/../vendor/rails/railties/lib/initializer.rb:43:in `send'
from ./script/../config/../vendor/rails/railties/lib/initializer.rb:43:in `run'
from ./script/../config/../config/environment.rb:11
from /Library/Ruby/Site/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
from /Library/Ruby/Site/1.8/rubygems/custom_require.rb:31:in `require'
from ./script/../config/../vendor/rails/activesupport/lib/active_support/dependencies.rb:495:in `require'
from ./script/../config/../vendor/rails/activesupport/lib/active_support/dependencies.rb:342:in `new_constants_in'
from ./script/../config/../vendor/rails/activesupport/lib/active_support/dependencies.rb:495:in `require'
from ./script/../config/../vendor/rails/railties/lib/commands/servers/webrick.rb:52
from /Library/Ruby/Site/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
from /Library/Ruby/Site/1.8/rubygems/custom_require.rb:31:in `require'
from ./script/../config/../vendor/rails/activesupport/lib/active_support/dependencies.rb:495:in `require'
from ./script/../config/../vendor/rails/activesupport/lib/active_support/dependencies.rb:342:in `new_constants_in'
from ./script/../config/../vendor/rails/activesupport/lib/active_support/dependencies.rb:495:in `require'
from ./script/../config/../vendor/rails/railties/lib/commands/server.rb:39
from ./script/server:3:in `require'
from ./script/server:3

First off, read Raimonds Simanovskis's great post on how to setup Ruby and Oracle instant client on Snow Leopard, if you haven't already. Note that the environment variables are different for different operating systems (see Oracle documentation). You may get the same or similar error by not having an environment variable missing/misspelled/setup incorrectly.

If you get the error I mentioned above, you might have a problem with version 2.0.0 and higher of ruby-oci8 as noted by Brendan Boesen. Assuming you have everything else working, just uninstall ruby-oci8 and install the older version, compiling it for 64-bit Intel architecture via:


gem uninstall ruby-oci8
env DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH ARCHFLAGS="-arch x86_64" gem install ruby-oci8 --version "< 2.0.0"

After doing that, I tried ./script/server again, and it worked fine.

Update: October 27, 2010: With Rails 3 you can use Ruby-oci8 2.0.4, and Oracle Enhanced Adapter 1.3.1. See this post.