Mongodb for non-x86 cpus

Mongo for non-x86 cpus

Background

I like the interface/api of mongoDB but I want to run it on FreeBSD/sparc64. mongo is very x86 oriented in a number of ways:

It assumes a little endian processor
It assumes that alignment doesn't matter. E.g. on FreeBSD/sparc64, you get a bus error and a crash if the data isn't properly aligned in memory.
It assumes that chars are signed (not a problem on sparc, though)

Here are examples of architectures that differ in those aspects:

OS/Arch	FreeBSD/sparc64	debian-ppc	debian-armel	debian-mipsel
Endianess	big	big	little	little
Alignment important	Y	N	Y	Y
char	signed	unsigned	unsigned	signed

Summary of changes

Added header util/bswap.h
Found the packed disk structures with embedded little endian types. Added little<T>, which behaves approximately as a T, but is stored little endian.
In many places pointers or references to e.g. unaligned little endian integers were returned. little<T> can be directly casted to from a char* with little<T>::ref.
Added tests is SConstruct for sync_fetch_and_add, since the gcc I have doesn't define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4, but still has the function.
-Wcast-align, which discovers many places where there are problematic casts (will not produce warnings on x86, though)

I have tried to change the code outside of bswap.h and the bson-related files as little as possible. The code is not optimized at all (a little, couldn't help myself), but x86 should behave the same as before. Git repo at http://github.com/skrabban/mongo-nonx86.

TODO:

Fixup the master branch. v1.8 seems to be working ok on for example SPARC64, ARM and PowerPC.

Status 2012-04-17

Building master on FreeBSD after some trouble with linking.

Status 2011-08-11

Sparc machine is still doing some dur test. Mac OS X/ppc loops (forever?) on

m31200| Tue Aug  9 16:59:46 [migrateThread] warning: migrate commit attempt time d out contacting 2 slaves for 'test.foo' { _id: MinKey } -> { _id: 52.0 }

OTOH this also happens on plain mongo/v1.8 on Mac OS X/x86.

Cannot run tests on debian/ppc since xulrunner was updated. Cannot kill operations. Same on ubuntu/amd64 with plain mongo/v1.8.

Status 2011-08-08

Changed from master to the v1.8-branch. FreeBSD/Sparc64 smoke all fails in 3 tests more than FreeBSD/amd64 does on plain mongodb/v1.8. Hopefully this is corrected with 29fd8d2ff07f80ebc00c6a410a066a1e8b5b278c. Smoke all is currently running on FreeBSD/Sparc64 and Mac OSX/ppc. "./test" succeeds on the following platforms: debian/mipsel, debian/ppc, debian/armel, FreeBSD/sparc64, MacOSX/ppc, linux/x86.
Status 2010-11-15
Pulled from mongo master. Unhappily found the new appendStruct function in builder.h.
smoke.py (FreeBSD 8.1/sparc64)
460 tests succeeded 9 tests didn't get run The following tests failed (with exit code): /usr/home/pi/mongo/jstests/slowWeekly/update_yield1.js 253 /usr/home/pi/mongo/perftest 17 /usr/home/pi/mongo/jstests/parallel/shellfork.js 253 /usr/home/pi/mongo/jstests/slowWeekly/query_yield2.js 253 /usr/home/pi/mongo/jstests/slowWeekly/indexbg1.js 253 /usr/home/pi/mongo/jstests/slowNightly/sharding_balance4.js 253 /usr/home/pi/mongo/jstests/parallel/allops.js 253 /usr/home/pi/mongo/jstests/parallel/del.js 253 /usr/home/pi/mongo/jstests/perf/find1.js 253 /usr/home/pi/mongo/jstests/slowWeekly/query_yield1.js 253 /usr/home/pi/mongo/jstests/slowNightly/btreedel.js 253 /usr/home/pi/mongo/jstests/parallel/repl.js 253 /usr/home/pi/mongo/jstests/parallel/insert.js 253 /usr/home/pi/mongo/jstests/parallel/basicPlus.js 253 /usr/home/pi/mongo/jstests/parallel/basic.js 253 /usr/home/pi/mongo/jstests/parallel/manyclients.js 253

Status 2010-11-13
OID:s are now big endian also on big endian machines (so the OID:s increasing test should work). stat-counters are aligned.
Status 2010-11-08
stats/counters converted so that they are atomic. The BSONObj returned has to be created new everytim.
TODO: copyBE and readBE for geo and OIDs.
Status 2010-10-27
smoke.py (FreeBSD 8.1/sparc64)
455 tests succeeded 9 tests didn't get run The following tests failed (with exit code): /usr/home/pi/mongo/jstests/slowWeekly/update_yield1.js 253 /usr/home/pi/mongo/perftest 17 /usr/home/pi/mongo/jstests/parallel/shellfork.js 253 /usr/home/pi/mongo/jstests/slowWeekly/query_yield2.js 253 /usr/home/pi/mongo/jstests/slowWeekly/indexbg1.js 253 /usr/home/pi/mongo/jstests/slowNightly/sharding_balance4.js 253 /usr/home/pi/mongo/jstests/parallel/repl.js 253 /usr/home/pi/mongo/jstests/parallel/allops.js 253 /usr/home/pi/mongo/jstests/parallel/del.js 253 /usr/home/pi/mongo/jstests/perf/find1.js 253 /usr/home/pi/mongo/jstests/sharding/auto2.js 253 /usr/home/pi/mongo/jstests/slowWeekly/query_yield1.js 253 /usr/home/pi/mongo/jstests/parallel/basic.js 253 /usr/home/pi/mongo/jstests/parallel/insert.js 253 /usr/home/pi/mongo/jstests/sharding/auto1.js 253 /usr/home/pi/mongo/jstests/slowNightly/sharding_passthrough.js 253 /usr/home/pi/mongo/jstests/parallel/basicPlus.js 253 /usr/home/pi/mongo/jstests/objid6.js 253 /usr/home/pi/mongo/jstests/parallel/manyclients.js 253

Current status 2010-10-21
sparc: test runs with 0 failures.
Current status 2010-10-20
Segfault in btreetests since update from master. x86 seems ok.
Current status 2010-10-19
test runs with 0 failures on FreeBSD 8.1/sparc64 and debian armel in qemu. test runs with 1 failure on debian powerpc in qemu.
Per Ola Ingvarsson pi at df.lth.se