Canvas auto panning based on input focus

[erikos]

More info: ​http://wiki.sugarlabs.org/go/Features/Touch/Development#Canvas_auto_panning_based_on_input_focus

Current patch for GTK+: ​http://dev.laptop.org/~erikos/gtk3_patches/0003-OSK-im-osk-position.patch

toolkit-gtk3 bit that landed: ​http://git.sugarlabs.org/sugar-toolkit-gtk3/sugar-toolkit-gtk3/commit/b44a235d9fca562522ea02832424c37bf30185e7

[garnacho]

last iteration of the gtk+ patch, meant to work on webkit too

[garnacho]

This last patch deviates from the branch that's upstream. Unlike the patches there, it makes every IM-aware widget trigger upwards OSK-triggered resizes (including WebKitWebView and AbiWidget) with no additional changes, but does so by breaking the conceptual barrier between GtkIMContext and GtkWidget (the former isn't aware of the latter, mostly a design detail). This hack would only break (eg. not propagate resizes) silently if the IM context user GdkWindow does not pertain to a GtkWidget, AFAIK there's no such case

[erikos]

I have two gtk3 rpms with the different patches, the one from the branch is ​http://dev.laptop.org/~erikos/gtk_rpms_os9/ and the second one at ​http://dev.laptop.org/~erikos/gtk_rpms_os9/osk2/

I have added a modified ​activity.py file from helloworld that has a Textview with some text. For debugging purposes there is as well ​the window.py from the toolkit-gtk3 with some debug output.

[erikos]

Both patches do not work for me. When I click in an area that would be covered by the OSK, the scrolling is not done to keep that area visible. Some scrolling does happen but seems to be just due to the scrolling area itself.

Looking at the logs we do not receive the 'request-clear-area' signal, seen with both patches.

[erikos]

Thanks for the patch, included in ​http://arm.koji.fedoraproject.org/koji/taskinfo?taskID=1243931

[erikos]

Ok, I added the new rpm to my public repo, so it will be in the next build (os11). From a quick test, things get into the right direction.

We need to handle as well the case when we get to the bottom of a document. There has to be osk-height more space so you can get as well to the bottom. Not sure yet where this compensation code goes.

[erikos]

Test cases:

• Terminal #4151

• Browse ​http://bugs.sugarlabs.org/ticket/4151#comment:3

[godiard]

Tested in os11, this change do not solves all the issues.

We need resize the canvas area to use only the visible area, between the toolbar and the osk,
if not, all the content in the bottom will not be accessible. Can be seen in Terminal, Write or Paint.

Adding log in window.py request_clear_area_cb method, this method is never called, in Terminal or Write activities.
garnacho say: garnacho> gonzalo, it's a signal that propagates from the bottommost to the topmost widget, other widgets may intercept it, eg. a scrolledwindow

I have tried catch the signal in other widgets without success.

I have tried to use the 'visibility-notify-change', on the canvas widget, and we can get a information about if the canvas is GDK_VISIBILITY_UNOBSCURED or GDK_VISIBILITY_PARTIAL, but I couldn't find how get the size of the real area obscured by the osk. Tried different ways, (the clipped area by cairo in the draw event, the allocated size by the widget) but all have the information like if the widget is visible behind the osk using all the available size.

I don't know if is available in any way, but is frustrating we don't have a simple api to solve simple questions like: is the osk visible? how much is the height/width? and commands to open/close on demand.

[garnacho]

Updated OSK patch, scrolled window now works if hitting the document end

[garnacho]

textview patch, update handles on relocation after request-clear-area

[erikos]

Thanks for the new patch Carlos. I did test with my modified HelloWorld activity, which has a scrolled window, textview inside. When I tap on text at the bottom the text to edit is moved up, right above the OSK. The cursor does as well update the position accordingly (it is over the OSK though).

If I hit enter, the current text gets under the OSK. Another issue is that if I scroll in that situation everything gets messed up in regards to the position. A regression for the scrolled window, is that when overshooting the widget does not take up the space again. A border is left.

The terminal and Browse use case both do not work with your patch. Is that expected. Anything we have to change here? Which use case do you test with?

[garnacho]

Replying to erikos:

If I hit enter, the current text gets under the OSK. Another issue is that if I scroll in that situation everything gets messed up in regards to the position. A regression for the scrolled window, is that when overshooting the widget does not take up the space again. A border is left.

I'm looking into fixing these

The terminal and Browse use case both do not work with your patch. Is that expected. Anything we have to change here? Which use case do you test with?

The VTE terminal widget has built-in scrolling and does not use GtkScrolledWindow, if Browse uses WebKit2 nowadays that might be the case too, I heard some months ago they were planning to do their own scrolling implementation

[dsd]

We don't use WebKit2 yet because of its lacking touch support in the scrolled window. At the moment Browse uses a regular scrolled window.

[erikos]

Replying to garnacho:

Replying to erikos:

If I hit enter, the current text gets under the OSK. Another issue is that if I scroll in that situation everything gets messed up in regards to the position. A regression for the scrolled window, is that when overshooting the widget does not take up the space again. A border is left.

I'm looking into fixing these

The terminal and Browse use case both do not work with your patch. Is that expected. Anything we have to change here? Which use case do you test with?

The VTE terminal widget has built-in scrolling and does not use GtkScrolledWindow, if Browse uses WebKit2 nowadays that might be the case too, I heard some months ago they were planning to do their own scrolling implementation

No, Browse uses WebKit. We do add the WebView to a scrolledwindow. In the activity window we do not get the 'request-clear-area' signal in both cases. I just removed the scrolledwindow in Browse, no signal neither.

[garnacho]

Updated patch, fixes some issues

[garnacho]

There's a few issues I could see here:
1) The maliit IM context emits the osk signal far too many times, and on all imcontext around, not just on the focus widget's
2) Input method usage in the VTE terminal widget and WebKit don't fully behave like GtkTextView's or GtkEntry's, this was the reason the clear-request-area signal wasn't being emitted properly on Terminal or Browse
3) There are some intrinsic synchronicity issues on the handling of focus-in, button-press, button-release and request-clear-area, hence some funkyness when the scrolledwindow leaves place for the osk

The last patch I submitted does what as much as possible on 1, fixes 2 and partly fixes 3

[erikos]

Thanks Carlos for investigating. I am building now with the new patch:

​http://arm.koji.fedoraproject.org/koji/taskinfo?taskID=1256251

[erikos]

Test case for Browse:

​http://dev.laptop.org/~erikos/textarea.htm

[erikos]

There are two cases in the shell where we want this to work: the Journal list view and the detail view.

In the Journal list view this is mostly working. When you tap on the title of an entry that is below the area where the OSK would cover the list does scroll up after a while. The title is half visible. This is as well the behaviour of items at the bottom of the scrolled window.

In the detail view we do not have a scrolled window. We could maybe add there one (that would only be visible and scroll when needed) to get this functionality.

[garnacho]

Updated patch

[garnacho]

This patch solves multiple oddities in scrolledwindow, and has better interaction with WebKit webviews, there's still a bug in WebKit though where the cursor coordinates it reports to the IM context do not match with the coordinate space of the client GdkWindow given to the IM, so depending on the scrolling amount this works better or worse in Browse

[garnacho]

Updated patch, has some necessary changes to implement Write logic

[garnacho]

Write patch, implement OSK resize in a subclass of Abi.Widget

[garnacho]

With these patches the textview and Write cases work nicely here, for Browse there still the WebKit bug where the coordinates don't match with the GdkWindow provided to the input method, looking into making a patch for this

[erikos]

rpm with new patch: ​http://arm.koji.fedoraproject.org/koji/taskinfo?taskID=1261450

[erikos]

I did a new ​rpm and removed the patch from #4060 since that broke scrolling in general.

Write does still not work with the OSK. There is a black part below where the cursor is. Also the title editing in the Journal stopped working.