Well, here's the entry I wanted to post last night, when Lori was using the net-connected computer. (I sigh that none of the three LJ clients I've used support disconnected operation. And as a side note, I note that an email-to-LJ gateway would solve this and a bunch of other problems. Perhaps I'll get amazingly inspired and write one.)
In the continuing story of the quest for compatibility:
Today's meeting turned into another long discussion about my efforts for compatibility. The upshot: Mike proposed again that instead of adding these nifty new features to support the use of unrecognized terms, we should limit ourselves to attested terms for this release. The purpose, of course, is to limit our risk exposure; we can't afford for this to slip much more.
He's definitely right. It's time to trim things back to a manageable state. But I feel so close to making this work...
So after the morning meeting, I continued debugging the new and improved system a bit more. (Mark said, "I understand the importance of being on a debugging roll. Keep going as long as you're on a roll, then switch over to backing things out when your roll ends." Very reasonable of him.) I found a major bug that I had introduced: an assertion was failing in the bowels of the code that reset ProfileSet statistics. This was partially hidden by the fact that the reset transaction was catching that assertion failure and proceeding blithely on. Oops.
So I fixed that. And then I said, "This time, for sure!" (sotto voce: "Aw, Bullwinkle, that trick never works!") and ran the tests again. I told Mark that I would bring home the laptop; if the run failed, I would back out the changes to a more stable state, make the necessary changes to the tree, and and demonstrate the more limited form of compatibility.
We were having Lori's parents over to celebrate Lori's mother's birthday, and I kept sneaking off to look at the progress of my changes.
Around 10pm, I got to look at the results in earnest. The run failed. But it had far fewer errors. And I looked at the errors and tried to debug them a little. It turned out that I wasn't including the TermLexicon from the old morpher in the new morpher. Oops. And that would take only about an hour to test...
This is so foolish of me. I ought to just be backing out the changes--I may not have time anymore to make all the changes and do a successful run, even if I pass up sleep. And honestly, there's no basis in experience to believe it'll work this time. It's like saying, "Gee, the Coyote has really been learning from experience, and he's got a really great plan--I bet he'll catch the Roadrunner this time." A conceivable proposition, but not one you can make book on.
"Ralph Melton, Suuuper-Genius. I like the sound of that."
I was depressed. I feared that I had doomed myself by pressing forward when I should have backed up.
The TermLexicon fix helped things a lot. I reran the retrieval test before I went to bed to see if that cleared any more discrepancies up. It was too late for me to back up--if it was necessary for me to back up, I had lost a day by fighting matters forward.
When I looked at things in the morning, things were down to nine serious discrepancies out of 140,000 documents.
I took the laptop into the insurance agent's office and debugged code while I waited for the agent. (More on that later, I hope.) I got in to the office at noon and continued debugging before going to lunch. The scores were matching--it looked like another thresholding problem. And lo and behold, the maximum score computation for retrieval wasn't taking TFs into account. And with that fix, it looked like all those discrepancies were explained.
I made that fix and started a run while I went out to lunch with Mike. When I got back and checked the results, it turned out that there was only one discrepancy--and that turned out to be explained by a problem we'd noted and found a workaround for before. Mark agreed that it wasn't necessary to fix the tree and rerun both sides of the test to prove that we could eliminate that last discrepancy.
Against my own expectations, I had achieved that goal that had been just out of reach for so long. I had caught the Roadrunner.
I felt that I ought to be doing some sort of elated little victory dance, but I felt more drained than elated. I did take the time to mention the good news to all three co-workers in the building, though.
Then I went on to test that my recent changes hadn't broken anything in our other test cases. The first went swimmingly; everything was as expected. The second, though, failed dramatically--differences in every document. As I started to debug that, I found another bug I'd introduced that was causing assertion failures. But as I worked on debugging that, I found that I was using the wrong tree for one of the runs--all those discrepancies were due to a single major error in how I was running the things.
I started the overnight regression tests and went home with a sense of relief.