migration.migrate_from_0 fails with file not found error

[carrott]

os300py Dextrose on XO-1

Migrating a Journal from build 767 (8.2.0) fails with a file not found error.

Attached patch creates the missing directory.

also, migration doesn't seem to go completely smoothy even with this patch. Migrated entries have a black box with 100% in it.

Should there be additionl migration code from version 1 to version 5?

[bernie]

Andres, could you give a look at this bug?

Carrot, what happens if you try to migrate to an older build with Sugar 0.84? I'd like to figure out which migration step screws up.

[alsroot]

carrott: will be useful to move os.makedirs(layoutmanager.get_instance().get_metadata_path(uid))
to migrate_from_0() itself since all _migrate* functions need uid directory. Also, please prepare your patch w/ "git format-patch" command to preserve your authorship.

[carrott]

Regarding which step screws up, as far as I can tell there is only one step, that performed in migration_from_0(). This seems surprising since the current journal version is version 5. The migration_from_0() doesn't seem to properly migrate the journal. The shell.log (attached) is full of exceptions.

Date parsing goes wrong, corrupting the self._cached_row

Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/jarabe/journal/listmodel.py", line 153, in on_get_value
    ctime = time.mktime(time.strptime(ctime, CTIME_FORMAT))
  File "/usr/lib/python2.6/_strptime.py", line 454, in _strptime_time
    return _strptime(data_string, format)[0]
  File "/usr/lib/python2.6/_strptime.py", line 325, in _strptime
    (data_string, format))
ValueError: time data dbus.ByteArray('1282443424', variant_level=1) does not match format '%Y-%m-%dT%H:%M:%S'

Access to the corrupt self._cached_row then fails

IndexError: list index out of range
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/jarabe/journal/listmodel.py", line 124, in on_get_value
    return self._cached_row[column]

I would suggest making the self._cache_row calculations failure atomic and/or more robust to bad data.

The journal itself shows a strange progress bar with 0% (see attached screenshot) which remains even when you re-open an entry.

If anyone needs example journals, I have about 20 from os767, but they have personal photos and stories so I don't think I should post them here.

[carrott]

shell.log showing date parsing failure and subsequent errors

[carrott]

journal screenshot showing 0% progress bar

[carrott]

I tried a manual restore on os802 (aka 8.2.1) by replacing the .sugar/default/datastore directory and ctrl-alt-erase restarting. This proceeded with no errors and the journal doesn't have the 0% progress bar thing.

However, this version doesn't seem to have a migration, so maybe the journal format hasn't changed. I'll try os850.

[carrott]

os851 doesn't have a restore from journal option. As for 802, I replaced the .sugar/default/datastore directory and ctrl-alt-erase restarted. This seemed to work a lot better than os373pyg's migration, but there were a still a number of warnings in the datastore log, see os851-datastore.log attached.

1282458495.433503 WARNING root: Index updating, returning all entries
1282458502.002427 WARNING root: Unknown term(s): dbus.Dictionary({dbus.String(u'order_by'): dbus.Array([dbus.String(u'-mtime')], signature=dbus.Signature('s'), variant_level=1), dbus.String(u'limit'): dbus.Int32(55, variant_level=1)}, signature=dbus.Signature('sv'))

[carrott]

datastore log from os851 after restoring a journal from os767

[sascha_silbe]

Can you attach a tarball of a 767 data store, please? The oldest I have is from 802, so that's all I can test against. Even better would be two tarballs: an "artificial" one that I can distribute freely and a copy of a "real-life" data store that was in regular use. I'll keep the latter one confidential; you could send it via email (silbe sugarlabs org), scp it to sunjammer or upload it somewhere.

To create a tarball, please issue the following command (all in one line) in Terminal:

tar -czf datastore-backup-"$(cat /ofw/serial-number)"-$(date -I).tar.gz -C ~/.sugar/default datastore

[carrott]

datastore.tar.gz prepared by me using build 767

[alsroot]

I've commited mkdir patch
 http://git.sugarlabs.org/projects/sugar-datastore/repos/mainline/commits/efa47c462ccae70e6f54ad5f5308b533aabd2476

[sascha_silbe]

Replying to carrott:

os300py Dextrose on XO-1

Migrating a Journal from build 767 (8.2.0) fails with a file not found error.

I can't reproduce this using the test suite (based on sugar-datastore 0.89) with any of the data stores you gave me and the datastore.log attached to this ticket doesn't show any error either.
Are you able to reproduce this with os300py using any of the data stores? If so, which one and what exactly is the error message (complete datastore.log preferred)?

Attached patch creates the missing directory.

While alsroot already pushed a patch, I'd like to understand why it failed at all. From looking at the code, this error should be impossible.

also, migration doesn't seem to go completely smoothy even with this patch. Migrated entries have a black box with 100% in it.

That sounds like #2208, so let's handle it there as it doesn't seem to be related to the mkdir issue.

Should there be additionl migration code from version 1 to version 5?

No, we always migrate from some old version to the very latest version. There are no intermediate steps.

Thanks for the attached data store! May we redistribute it (as part of the test suite) under the terms of GPL version 2 and later? When attributing you (in the git log), should I use your real name + email address or just your user name (carrott)?

[carrott]

I can't reproduce this using the test suite (based on sugar-datastore 0.89) with any of the data stores you gave me and the datastore.log attached to this ticket doesn't show any error either

Sorry, the datastore.log is from os851 which seems to migrate successfully. I included it because there were some other strange errors.

I'll look at why you can't reproduce the missing directory error tomorrow and post a proper bug report with steps to reproduce -- this one was an "almost no internet special" report. Sorry. I did have a fever (from lack of internet no doubt) while coding the fix, but I reproduced it many times while fixing it, so I'm sure it's in there somewhere.

Should there be additionl migration code from version 1 to version 5?

No, we always migrate from some old version to the very latest version. There are no intermediate steps.

So if you have one migration method that migrates from the oldest to the newest, what happens when you encounter an intermediate version, one that isn't the oldest but isn't the newest? Do you still run the same migration code? I was expecting some sort of chain of methods 0->1 followed by 1->2 and 2->3, or am I just not looking in the right place?

Thanks for the attached data store! May we redistribute it (as part of the test suite) under the terms of GPL version 2 and later? When attributing you (in the git log), should I use your real name + email address or just your user name (carrott)?

Sure, real name + email is ok.

[carrott]

I can't reproduce this using the test suite (based on sugar-datastore 0.89)

I was able to reproduce this problem with os373pyg. See attached os373pyg-datastore.log

* make a directory on a usb flash drive /backup/<serial number>
* copy the datastore.tar.gz (created on os767) attached here into that directory
* close all activities
* open the journal and restore from usb
* only two entries will be restored into the journal and they will have a 0% progress bar
* many errors will be recorded in the datastore log (see attached os373pyg-datastore.log)

The datastore.tar.gz journal backup attached has a photo from record, a picture from paint, a story from write. None of these were successfully restored.

[carrott]

datastore.log from os373py after restoring a journal from os767

[sascha_silbe]

The line numbers don't match with the code in the sucrose-0.84 branch. Bernie, where's the git repo with the code and what commit corresponds to os373py?

[bernie]

Replying to sascha_silbe:

The line numbers don't match with the code in the sucrose-0.84 branch. Bernie, where's the git repo with the code and what commit corresponds to os373py?

Dextrose is based on sucrose-0.88, not 0.84.

There's no git repo for the sugar modules. To build the custom rpms, we use the release tarballs plus about 100 patches:  http://wiki.sugarlabs.org/go/Dextrose/Build_System

[sascha_silbe]

I don't see an easy way to recreate the exact source code that triggered this bug and I don't have time to dive into your build process.
As I have neither been able to reproduce it on my side and nor to get the source code for your side, I cannot continue diagnosing this bug. Sorry.

[bernie]

Hopefully this bug will disappear when we replace the current patch series with backported code from 0.90.

Thus, reassigning this bug to myself.

[alsroot]

Checked attached datastore.tar.gz in new sugar env with datastore-0.90, works fine w/o any errors.