Opened 9 years ago

Closed 6 years ago

#2149 closed defect (fixed)

migration.migrate_from_0 fails with file not found error

Reported by: carrott Owned by: bernie
Priority: Unspecified by Maintainer Milestone: Unspecified
Component: Sugar Version: 0.88.x
Severity: Critical Keywords: migration dextrose r!
Cc: dfarning@…, aa, bernie, alsroot Distribution/OS: Fedora
Bug Status: Unconfirmed

Description

os300py Dextrose on XO-1

Migrating a Journal from build 767 (8.2.0) fails with a file not found error.

Attached patch creates the missing directory.

also, migration doesn't seem to go completely smoothy even with this patch. Migrated entries have a black box with 100% in it.

Should there be additionl migration code from version 1 to version 5?

Attachments (6)

migration.patch (652 bytes) - added by carrott 9 years ago.
shell.log (270.7 KB) - added by carrott 9 years ago.
shell.log showing date parsing failure and subsequent errors
Screenshot of _Journal__1.png (78.9 KB) - added by carrott 9 years ago.
journal screenshot showing 0% progress bar
os851-datastore.log (5.7 KB) - added by carrott 9 years ago.
datastore log from os851 after restoring a journal from os767
datastore.tar.gz (240.8 KB) - added by carrott 9 years ago.
datastore.tar.gz prepared by me using build 767
os373pyg-datastore.log (2.5 KB) - added by carrott 9 years ago.
datastore.log from os373py after restoring a journal from os767

Download all attachments as: .zip

Change History (24)

Changed 9 years ago by carrott

comment:1 Changed 9 years ago by bernie

  • Cc aa added
  • Keywords dextrose added

Andres, could you give a look at this bug?

Carrot, what happens if you try to migrate to an older build with Sugar 0.84? I'd like to figure out which migration step screws up.

comment:2 Changed 9 years ago by tomeu

  • Component changed from sugar to sugar-datastore
  • Owner changed from tomeu to alsroot

comment:3 Changed 9 years ago by alsroot

  • Keywords r! added

carrott: will be useful to move os.makedirs(layoutmanager.get_instance().get_metadata_path(uid))
to migrate_from_0() itself since all _migrate* functions need uid directory. Also, please prepare your patch w/ "git format-patch" command to preserve your authorship.

comment:4 Changed 9 years ago by carrott

Regarding which step screws up, as far as I can tell there is only one step, that performed in migration_from_0(). This seems surprising since the current journal version is version 5. The migration_from_0() doesn't seem to properly migrate the journal. The shell.log (attached) is full of exceptions.

Date parsing goes wrong, corrupting the self._cached_row

Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/jarabe/journal/listmodel.py", line 153, in on_get_value
    ctime = time.mktime(time.strptime(ctime, CTIME_FORMAT))
  File "/usr/lib/python2.6/_strptime.py", line 454, in _strptime_time
    return _strptime(data_string, format)[0]
  File "/usr/lib/python2.6/_strptime.py", line 325, in _strptime
    (data_string, format))
ValueError: time data dbus.ByteArray('1282443424', variant_level=1) does not match format '%Y-%m-%dT%H:%M:%S'

Access to the corrupt self._cached_row then fails

IndexError: list index out of range
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/jarabe/journal/listmodel.py", line 124, in on_get_value
    return self._cached_row[column]

I would suggest making the self._cache_row calculations failure atomic and/or more robust to bad data.

The journal itself shows a strange progress bar with 0% (see attached screenshot) which remains even when you re-open an entry.

If anyone needs example journals, I have about 20 from os767, but they have personal photos and stories so I don't think I should post them here.

Changed 9 years ago by carrott

shell.log showing date parsing failure and subsequent errors

Changed 9 years ago by carrott

journal screenshot showing 0% progress bar

comment:5 Changed 9 years ago by carrott

I tried a manual restore on os802 (aka 8.2.1) by replacing the .sugar/default/datastore directory and ctrl-alt-erase restarting. This proceeded with no errors and the journal doesn't have the 0% progress bar thing.

However, this version doesn't seem to have a migration, so maybe the journal format hasn't changed. I'll try os850.

comment:6 Changed 9 years ago by carrott

os851 doesn't have a restore from journal option. As for 802, I replaced the .sugar/default/datastore directory and ctrl-alt-erase restarted. This seemed to work a lot better than os373pyg's migration, but there were a still a number of warnings in the datastore log, see os851-datastore.log attached.

1282458495.433503 WARNING root: Index updating, returning all entries
1282458502.002427 WARNING root: Unknown term(s): dbus.Dictionary({dbus.String(u'order_by'): dbus.Array([dbus.String(u'-mtime')], signature=dbus.Signature('s'), variant_level=1), dbus.String(u'limit'): dbus.Int32(55, variant_level=1)}, signature=dbus.Signature('sv'))

Changed 9 years ago by carrott

datastore log from os851 after restoring a journal from os767

comment:7 Changed 9 years ago by sascha_silbe

  • Severity changed from Major to Critical

Can you attach a tarball of a 767 data store, please? The oldest I have is from 802, so that's all I can test against. Even better would be two tarballs: an "artificial" one that I can distribute freely and a copy of a "real-life" data store that was in regular use. I'll keep the latter one confidential; you could send it via email (silbe sugarlabs org), scp it to sunjammer or upload it somewhere.

To create a tarball, please issue the following command (all in one line) in Terminal:

tar -czf datastore-backup-"$(cat /ofw/serial-number)"-$(date -I).tar.gz -C ~/.sugar/default datastore

Changed 9 years ago by carrott

datastore.tar.gz prepared by me using build 767

comment:9 in reply to: ↑ description Changed 9 years ago by sascha_silbe

Replying to carrott:

os300py Dextrose on XO-1

Migrating a Journal from build 767 (8.2.0) fails with a file not found error.

I can't reproduce this using the test suite (based on sugar-datastore 0.89) with any of the data stores you gave me and the datastore.log attached to this ticket doesn't show any error either.
Are you able to reproduce this with os300py using any of the data stores? If so, which one and what exactly is the error message (complete datastore.log preferred)?

Attached patch creates the missing directory.

While alsroot already pushed a patch, I'd like to understand why it failed at all. From looking at the code, this error should be impossible.

also, migration doesn't seem to go completely smoothy even with this patch. Migrated entries have a black box with 100% in it.

That sounds like #2208, so let's handle it there as it doesn't seem to be related to the mkdir issue.

Should there be additionl migration code from version 1 to version 5?

No, we always migrate from some old version to the very latest version. There are no intermediate steps.

Thanks for the attached data store! May we redistribute it (as part of the test suite) under the terms of GPL version 2 and later? When attributing you (in the git log), should I use your real name + email address or just your user name (carrott)?

comment:10 Changed 9 years ago by carrott

I can't reproduce this using the test suite (based on sugar-datastore 0.89) with any of the data stores you gave me and the datastore.log attached to this ticket doesn't show any error either

Sorry, the datastore.log is from os851 which seems to migrate successfully. I included it because there were some other strange errors.

I'll look at why you can't reproduce the missing directory error tomorrow and post a proper bug report with steps to reproduce -- this one was an "almost no internet special" report. Sorry. I did have a fever (from lack of internet no doubt) while coding the fix, but I reproduced it many times while fixing it, so I'm sure it's in there somewhere.

Should there be additionl migration code from version 1 to version 5?

No, we always migrate from some old version to the very latest version. There are no intermediate steps.

So if you have one migration method that migrates from the oldest to the newest, what happens when you encounter an intermediate version, one that isn't the oldest but isn't the newest? Do you still run the same migration code? I was expecting some sort of chain of methods 0->1 followed by 1->2 and 2->3, or am I just not looking in the right place?

Thanks for the attached data store! May we redistribute it (as part of the test suite) under the terms of GPL version 2 and later? When attributing you (in the git log), should I use your real name + email address or just your user name (carrott)?

Sure, real name + email is ok.

comment:11 Changed 9 years ago by carrott

I can't reproduce this using the test suite (based on sugar-datastore 0.89)

I was able to reproduce this problem with os373pyg. See attached os373pyg-datastore.log

  • make a directory on a usb flash drive /backup/<serial number>
  • copy the datastore.tar.gz (created on os767) attached here into that directory
  • close all activities
  • open the journal and restore from usb
  • only two entries will be restored into the journal and they will have a 0% progress bar
  • many errors will be recorded in the datastore log (see attached os373pyg-datastore.log)

The datastore.tar.gz journal backup attached has a photo from record, a picture from paint, a story from write. None of these were successfully restored.

Changed 9 years ago by carrott

datastore.log from os373py after restoring a journal from os767

comment:12 follow-up: Changed 9 years ago by sascha_silbe

  • Cc bernie added

The line numbers don't match with the code in the sucrose-0.84 branch. Bernie, where's the git repo with the code and what commit corresponds to os373py?

comment:13 in reply to: ↑ 12 Changed 9 years ago by bernie

Replying to sascha_silbe:

The line numbers don't match with the code in the sucrose-0.84 branch. Bernie, where's the git repo with the code and what commit corresponds to os373py?

Dextrose is based on sucrose-0.88, not 0.84.

There's no git repo for the sugar modules. To build the custom rpms, we use the release tarballs plus about 100 patches: http://wiki.sugarlabs.org/go/Dextrose/Build_System

comment:14 Changed 9 years ago by sascha_silbe

I don't see an easy way to recreate the exact source code that triggered this bug and I don't have time to dive into your build process.
As I have neither been able to reproduce it on my side and nor to get the source code for your side, I cannot continue diagnosing this bug. Sorry.

comment:15 Changed 9 years ago by bernie

  • Owner changed from alsroot to bernie
  • Status changed from new to assigned

Hopefully this bug will disappear when we replace the current patch series with backported code from 0.90.

Thus, reassigning this bug to myself.

comment:16 Changed 9 years ago by alsroot

  • Cc alsroot added

Checked attached datastore.tar.gz in new sugar env with datastore-0.90, works fine w/o any errors.

comment:17 Changed 6 years ago by dnarvaez

  • Component changed from sugar-datastore to Sugar

comment:18 Changed 6 years ago by dnarvaez

  • Resolution set to fixed
  • Status changed from assigned to closed
Note: See TracTickets for help on using tickets.