Coding done... halfway (in Off-topic)

Obscurans February 24 2008 3:28 AM EST

Yeah, truly off topic, but I finally worked the logical bugs out of my program. To the uninitiated, a logical bug is not the "error" messages your compiler throws at you, but the program being "correct" but not doing what you want.

The program's a prune tree for finding Costas arrays, checking conditions on placing a candidate. Very esoteric topic and highly mathematical.

Basically, a Costas array is n dots placed on a n by n grid such that each row and column contains exactly one dot, and the distance between each pair of dots is distinct. So
is valid, but
is not since the "distance" from the dots in row 1-2 is the same as from row 3-4.

Finding them is a major pain in (wherever you hurt most). They have a mathematical construction using finite fields for prime powers and I think up to 5 less than them. Except, that is spotty coverage and they haven't proved the existence of Costas arrays of size 32 and 33 (the smallest holes).

Now, 32 sounds small, except ignoring the distance condition, then there are exactly the number of permutations of 32 points of candidates or 32! = 263130836933693530167218012160000000. Yes, it's horrendous.

Using a row and column mask that is to be evolved, but that's to be coded, and another field entirely (evolutionary computation).

And by the way, somehow people have come up with ways to use these arrays in sonar and radar. So stone-hard math, used in... the military?

Obscurans February 24 2008 3:29 AM EST

That was after 10 hours and 3 resets. The evolutionary code is going to be a breeze after that...

Obscurans February 24 2008 6:03 AM EST

W00t, all the coding done, now heading over to cluster and hurling code at it for the next ~2 weeks...

Levon [Clocked Out] February 24 2008 6:08 AM EST

as funny as it sounds you actually got my mouth watering :p

AdminNightStrike February 24 2008 6:11 AM EST

What language are you writing in? Also, have you tested your program on small orders of N? For instance, you should be able to brute force N=15 in about 1 second on any modern computer.

Obscurans February 24 2008 6:21 AM EST

Yes, and the plan here is, since doing it for n=32 straight is going to... kill, I'm going a different route.

The ordinary prune tree would search each column from left to right and each row top to bottom (well, the simplest coding). I'm putting a row mask and column mask to change the order presented to the tree. Obviously that can change what limbs are chopped off, and having bigger limbs chopped off earlier helps with it.

So now, I can't make up a "perfect" mask without already solving the problem, so I'm using the evolutionary algorithm on the masks to minimize time-to-solution, for small n.

Then a prayer to the god of scaling, and test that solution for progressively higher n. The contents of the prayer, is, of course that there will be some sort of useful pattern that emerges out of that mess, that still works in higher n.

That problem is notorious for NOT scaling though, and my eh, supervisor, who is I think the 2nd out of 5 people in the world working on that problem, has all these earlier difficulty estimates (he's working on a classification of problems). And there are enormous jumps at n=12, 20 and so on. By enormous I mean above and beyond the expected factorial difficulty.

So, code's running and I already see bugs. That can't have been exposed by running dummy tests. The very evolved ones have a time-to-exhaustion of the entire search space (n^n) of... 1's and 2's. Probably sent the wrong data back after evaluation.

And interestingly, the data required for the masks is (n!)^(n+1), even more than the n! for the original problem. Another case of let's make the problem harder and backsolve into even harder versions of the original problem.

Obscurans February 24 2008 6:22 AM EST

Ah, straight C code. The fastest (and I don't write assembly). FORTRAN doesn't have all these pointy things, and I need them for hand-optimization.

I can brute force 20 on the identity mask (= no mask) by a couple of minutes.

TheHatchetman February 24 2008 6:27 AM EST

If ya don't mind me asking, what is all this for? :P

Obscurans February 24 2008 6:29 AM EST

Kicks, fun, a final project, and a 3% shot at a paper?

Obscurans February 24 2008 6:48 AM EST

Weirdest error I have seen to date: if I *slow* down the program by adding useless sleep()s, the answer is correct. If I don't, then the program runs in about a millisecond (which is wrong, since my 2.0G runs it for 10 seconds, and those Itaniums are 1.6G), and all the counts are <10.

Don't drink and drive, boys. I guess the sleeps stop the somehow-rampant parallelization? Problem is, that evaluation is not only not embarrassingly parallel, it's completely serial. Well almost, you can spread along limbs and come back.

AdminG Beee February 24 2008 6:58 AM EST

Cool, 5 years of playing CB and I've found myself confused when reading quite a few threads. This one takes the biscuit !

Can anyone tell me if this guy is breaking PG rules and needs a fine? I have no idea what the hell he's talking about...

QBBast [Hidden Agenda] February 24 2008 7:14 AM EST

I'm with Levon, totally drooling.

AdminLamuness February 24 2008 7:16 AM EST

From the initial look, it looks like this can be done in the order of O(n). Interesting problem though. I'll have to think it over.

Talion February 24 2008 7:21 AM EST

"Weirdest error I have seen to date: if I *slow* down the program by adding useless sleep()s, the answer is correct."

I ran across similar issues a good number of times during my programming career. Those are hardest one to figure out, the reason is never the same, and the problem is sometimes completely un-related to the code itself.

Good luck.

Obscurans February 24 2008 7:23 AM EST

Ah, but the permutation matrix condition makes it at least n! even when injecting that knowledge, and you have n(n-1)/2 extra binary exclusions (that overlap excessively). Finding one is at most O(n!)>O(e^n) by exhaustive trying. Prune tree search goes for the full n^n space, but chopping off limbs should make it somewhere near exponential, hopefully less.

G_Beee: chill and listen to codemonkey or similar :), and ignorance is totally work-safe.

Obscurans February 24 2008 7:25 AM EST

And icc is to blame, waste of money eh? gcc works fine, so I'll have to skip Intel's cool optimizations I guess. -O3 for the win. And it's not the optimizations on icc, just left in -lpthread (since I parallelize), and still shot.

Obscurans February 24 2008 7:55 AM EST

Ah, found the finite field construction.

Take (Zp*, *), the group of nonzero integers modulo prime p under multiplication. If g is a primitive root over Zp*, then the permutation {g^i}_i=1^p-1 is a Costas array.

g is primitive iff g^p = 1 and g^k != 1 for all k < p.

Note that this construction always puts a point at the lower-right corner, since by definition g^p = 1, so removing that point makes a Costas array of order p-2.

Et cetera, forgot the p-3 construction.

{cb1}Linguala February 24 2008 9:13 AM EST

Somebody give me a crash course math and programming...
This sounds familiar, but it's going so out of my league...
If only I was still a student, this would have made alot more sense.

BootyGod February 24 2008 9:31 AM EST

GB, I say ban him just to be safe ;P

I think he's communicating with his multi counterpart in a form we cannot hope to understand.

Burn the multi!


Obscurans February 24 2008 9:58 AM EST

Preliminary results~ W00t!

n / best-found length-to-complete-traversal

Evolved using population size 20, inverse number of moves-proportional selection and replacement with stillborn-able children, no crossover, n-point mutation for size n arrays, 1 million generations.

If anything, these numbers look... linear to me. Intercept goes somewhere ~3, but that's where there's even any meaning to the arrays.

Wizard'sFirstRule February 24 2008 10:14 AM EST

what is the definition of distance for this exercise? in the top diagram, the length between the first 2 dots and last 2 is the game, except on a different direction.

Obscurans February 24 2008 10:19 AM EST

Take them as vectors, so for:


so the difference for 1st and 2nd row is considered different from the 1st and 3rd. However 1st and 2nd row difference is exactly equal to that from the 3rd and 4th row, so that's not a Costas array.

i.e. (1,2) is different from (2,1).

Obscurans February 24 2008 10:20 AM EST

Results back for n=16, 2 runs, 413 and 417: still linear to me.

Little Anthony February 24 2008 10:23 AM EST

can I be your friend?

Obscurans February 24 2008 10:24 AM EST

Shure, go ahead :)

Obscurans February 24 2008 10:25 AM EST

And another run for size 16: 399. No ascending slope, not exponential.

Obscurans February 24 2008 10:51 AM EST

17: 450
18: 516 - from 18^18 = 39346408075296537575424

slope's higher, but not even

TheHatchetman February 24 2008 9:23 PM EST

seems like fun if that's what you're in to ^_^
I'm sufficient enough to "code" a html link. :P But this seems like a lot of stuff... what kind of computer are you using that can handle such a task?

Obscurans February 25 2008 9:00 PM EST

It's an ALTIX350 cluster.

 *                        _      ___                                  *
 *                       / \  |   |  | \/                             *
 *                      /---\ |_  |  | /\                             *
 *                                                                    *
 *       Welcome to the SGI Altix 350 at the U of Guelph              *
 *                                                                    *
 *                                                                    *
         This is an SGI Altix 350 system which consists of:
          32 Itanium2 Processors with clock speed 1.6GHz 
              Main memory size: 64 GB  Cache Size: 9 MB
                 SuSE Linux Enterprise Server 9 
                      Kernel 2.6.5-7.286-rtgfx
                         SGI Propack 4 SP3

 Type "helpme | more" for usage instructions ...

Roughneck February 25 2008 9:55 PM EST

1+1= 2 I am with you BGGG
This thread is closed to new posts. However, you are welcome to reference it from a new thread; link this with the html <a href="/bboard/q-and-a-fetch-msg.tcl?msg_id=002MRN">Coding done... halfway</a>