Category: Coding

I was happy to see that the JDBC 4.0 spec will contain support for improved BLOB and CLOB handling. Blob handling is still one of the most awkward areas of cross-database implementation work. It would be nice to have a consistent interface for blob creation, update, and streaming across databases. Ostensibly, that’s what we have in JDBC 3 . However, in practise, things are not so simple.

One of the fundamental issues is how the RDBMS handles LOB data. To perform an INSERT or UPDATE of blob data, the RDBMS hands the client back a pointer or “locator” to the blob data. The manner in which they do this varies across implementations and vendors. This means in practise that you have different limitations depending on which DBMS you are using. Oracle has always been a culprit (until version 10) with its infamous 4k limit on streaming data, which has resulted in hundreds of applications and frameworks (Hibernate among them) having to write Oracle-specific JDBC-level code to work around this issue.

We are currently working on a system where we have written a server component which accepts standard FTP connections from clients. The clients send large binary files via FTP, which are streamed and stored into the database within a single transaction. Various consistency checks are performed along the way, and if any of the checks fail, an error is returned to the client and the transaction is rolled back. This means that if the client receives a successful return, they can be sure that the data is stored and ready to be processed.

The main problem has been with trying to successfully stream the blob data in a clean and portable way. Our production database is Oracle, but it’s nice to be able to test on say, MySQL or HSQL. If you want to go straight to JDBC, you can use Oracle’s empty_blob() function, grab a blob locator from that, and then use the oracle.sql.BLOB type’s getBinaryOutputStream() to get an OutputStream to write to. You can then chunk the data from a socket directly to the database server. However, achieving this at a higher level is altogether more challenging. Thankfully, Spring has some streaming LOB support, which is what we have based our current solution on.

Continuing on from my last post about fixing some issues related to MySQL and Python on Solaris, I came across another issue this morning which also necessitated digging out a copy of good ole gdb. The basic issue was that CVSGraph segfaulted every time I attempted to generate a revision graph. This was reproducible every time, no matter what the input. Turning the verbosity to the maximum allowed level did not produce anything useful.

First thing I did was download a copy of gdb, and did a configure/make/make install. I initially thought that if I could get cvsgraph to produce a core dump on exit, I would be able to examine it within gdb and get some clues about the cause. However, I couldn’t get CVSGraph to automatically core dump, even after setting core policy using coreadm, as shown here.
UPDATE: I found the missing piece of the puzzle – the maximum core dump size had not been set via ulimit. Setting this enabled automatic core dumping.

The next step was to actually load cvsgraph into gdb and run a test session inside the debugger. After a couple of runs, I had isolated the problem to a specific routine. I set a breakpoint and ran through the test case again:

First, I set up the command line arguments:

(gdb) set args -c /usr/local/viewcvs-1.0-dev/cvsgraph.conf -r /usr/cvsroot cobra/build.xml,v

Then set a breakpoint at the relevant location:

(gdb) break cvsgraph.c:1092
Breakpoint 1 at 0x1311c: file cvsgraph.c, line 1092.

Then kick off the target program:

(gdb) run
Starting program: /root/cvsgraph-1.5.1/cvsgraph -c /usr/local/viewcvs-1.0-dev/cvsgraph.conf -r /usr/cvsroot cobra/build.xml,v

Once gdb hits the breakpoint, it stops and waits for instructions:

Breakpoint 1, expand_string (s=0x50d51 “d”, rcs=0x54450, r=0x545b8, rev=0x50770, prev=0x0, tag=0x0) at cvsgraph.c:1092
1092 t = mktime(&tm);

I manually stepped forward a couple of times until I hit the problem:

(gdb) n
1094 if(env)
(gdb)
1095 setenv(“TZ”, env, 1);
(gdb)

Program received signal SIGSEGV, Segmentation fault.
0xff3a0510 in memcpy () from /usr/platform/SUNW,Sun-Fire-V240/lib/libc_psr.so.1

Now we can do a stack backtrace to see where we were at the time:

(gdb) bt
#0 0xff3a0510 in memcpy () from /usr/platform/SUNW,Sun-Fire-V240/lib/libc_psr.so.1
#1 0x0001ed6c in setenv (name=0x1f4e0 “TZ”, value=0xffbfffac “GB”, replace=1) at ../../../libiberty/setenv.c:156
#2 0x00013140 in expand_string (s=0x50d51 “d”, rcs=0x54450, r=0x545b8, rev=0x50770, prev=0x0, tag=0x0) at cvsgraph.c:1095
#3 0x00016a28 in make_layout (rcs=0x54450) at cvsgraph.c:2937
#4 0x00019ecc in main (argc=327696, argv=0x50010) at cvsgraph.c:3879

So know we have isolated the problem down to setenv(), which is implemented in the GNU libiberty adapter library. I exited gdb and wrote a simple test case based on what CVSGraph was doing at the point in question, and found that the problem can be easily reproduced by calling putenv() to create an environment variable, and then immediately calling setenv() to reset the value. This may be due to a bug in the libiberty putenv implementation.

In reality, we dont really need the call to putenv() here – its actually redundant, as setenv() will allocate space for the new variable if necessary. So I simply commented out the offending line, remade CVSGraph, and now we have the (very useful IMHO) graphical branching display from ViewCVS.

I am currently in the process of setting up a continuous integration server, which involves installing packages like CVS, Ant, CruiseControl, Tomcat, Apache, MoinMoin, ViewCVS, and MySQL, and basically glueing them all together. It’s a relatively long-winded process (especially on Solaris, as it won’t necessarily have a lot of the GNU prerequisites installed). However, a lot of it is boilerplate and individual packages can be installed relatively quickly.

During the course of installing the various packages and prerequisites involved, you find that from time to time you need to make manual adjustments to makefiles, switch libraries or library versions, or alter the parameters passed to the C compiler, for instance. This is always where I am grateful that I have a pretty good knowledge of C-based development on Unix (rusty as it is), as it comes in very handy when installing lots of open-source onto a Unix box. However, one problem almost had me stumped.

Whilst trying to install the CVS query functionality that comes with ViewCVS, I needed to install MySQL-Python as a prerequisite. This is ostensibly a simple two-step procedure, python setup.py build and (as root) python setup.py install. However, it failed immediately, citing incorrect compiler flags. I located the point in setup.py where it sets those parameters (which it gets from mysql_config), and set them to more sensible parameters for Solaris. This compiled, but failed on the link stage with ld: warning: file /usr/local/mysql/lib/libmysqlclient.a(libmysql.o): wrong ELF class: ELFCLASS64. I tracked this down to a 64-bit MySQL version installed on a 32-bit platform. After uninstalling and reinstalling the correct version of MySQL, I attempted another build of MySQL-Python. This time, it compiled and linked fine , produced a shared library, and was duly installed.

Next, I tried to run a simple Python test script to verify that it was OK. This failed with:

File “/usr/local/lib/python2.3/site-packages/MySQLdb/__init__.py”,
line 27, in ?
import _mysql
ImportError: ld.so.1: /usr/local/bin/python: fatal: relocation error:
file /usr/local/lib/python2.3/site-packages/_mysql.so: symbol
mysql_errno: referenced symbol not found

This was altogether tougher to diagnose. I first thought that mysql_errno might be defined differently for the reentrant versus the non-reentrant MySQL client libraries, so I replaced "-lmysqlclient_r" with "-lmysqlclient" on the linker command line. This didn’t make any difference. I also explicitly added the MySQL library path into the LD_LIBRARY_PATH variable, which also made no difference.

Eventually, I found the answer (at least for my version of Solaris) – pass the shared lib flag to the linker explicitly – for some reason the installer command line doesn’t do this correctly. I found this out by running gcc manually, initially replicating the flags used by the Python installer.

The full gcc command line then looked like this:

gcc -I /usr/local/mysql/include/ -I /usr/local/include/python2.3/ -Wl,-G -fPIC -Wall _mysql.c -o _mysql.so `mysql_config --libs_r`


When the shared lib had built, I copied it to the appropriate spot in the Python build directory, and then just ran setup.py install to produce a working MySQL-Python client lib.

An easy way to fix this is to modify setup.py. Under the line that reads:

extra_compile_args = config(“cflags”)

Add the following lines:

extra_compile_args.append(“-fPIC”)
extra_compile_args.append(“-Wl,-G”)