~eliasnaur/gio#164: 
Proper modal/z-index/overlay support (proposal)

We have (as a community) talked several times about needing a better way to handle elements that overlap. The most recent conversation on the topic that I'm aware of is here in the #gioui slack channel.

As contributors like ~jackmordaunt and I continue to work on materials, we're running up against the lack of this feature pretty often.

~jackmordaunt recently came up with an interesting trick that (to me) seems to have the potential to address this space, so I'd like to resurrect the conversation around this topic.

The current API for widgets in gio returns only the dimensions of a rendered widget. The returned dimensions must be only the space occupied by the widget's normal (non-overlay) contents. This provides no information about any overlay content to higher-level render logic.

Jack's suggestion is to extend the widget interface so that it returns both the widget's occupied space in the default Z layer and the dimensions (and op.CallOp) of any overlaid content. These can be packed into a single structure or separated, it doesn't impact the core idea.

One implementation would look like:

type Overlay struct {
    layout.Dimensions
    op.CallOp
    ZIndex int // optional, this approach works without one
}

type Widget func(gtx layout.Context) (layout.Dimensions, Overlay)

The overlay type's CallOp is responsible for any offsetting relative to the widget that created it, so higher-level layout logic merely needs to understand that the content is overlaid and appropriately either lay it out itself or return it within its own overlay return type.

Each layout type can accept the Overlay returned by child widgets, apply appropriate transformations, combine multiple child widget overlay CallOps into one CallOp, and return it. In your main event loop, there can be a single function invoked to apply all of the overlay content, and this would be an ideal place to resolve Z-indicies if we introduced them into the Overlay type.

I have twisted Jack's original proposal some above, as he was resolving the overlay content within the nearest (upward) layout.Flex operation, but I think that deferring the application of the CallOp until the end of the render pass is more flexible. No idea what performance might look like though, so perhaps this is a terrible plan.

To view Jack's original proposal, see these materials patches. To try his working demo of the approach, see this branch of materials. Run the example program, navigate to the "Text Field Features", and try the dropdown select widgets. They utilize this approach.

Anyway, I just wanted to raise this discussion again in a forum where it won't get lost. Jack and I are both new to this kind of thing, and we may have overlooked some serious problems here. If this should have gone to the mailing list, I can send it there instead.

Status
REPORTED
Submitter
~whereswaldon
Assigned to
No-one
Submitted
9 months ago
Updated
a month ago
Labels
No labels applied.

~jackmordaunt 9 months ago

Thanks for the summary Chris.

If I may weigh in on api design:

I think it would be better to combined all data returned into a single struct, eg Render or RenderInfo, Data, et al.

The reason being that overlay data is optional, and most of the time the consumer of the the layout package wont care about overlay data, especially when composing high level layouts.

Conversely it's the widget implementation and the layout implemenations that will care about overlay data.

Using mutiple returns would required the consumer to specify all 3 types in the widget closure, for example:

layout.Rigid(func(gtx C) (D, O) {...})

Obviously if more data types are returned by the widget function, then the number of types in the signature grows linearly - creating a lot of noise.

layout.Rigid(func(gtx C) (D, O, T, B) {...})

It also means that any time types are added, all layout code will break and need to be refactored causing headaches downstream.

If the number of return types remains exactly one, then data can be added over time without breaking any code, and when data isn't used the consumer can ignore it. This captures the optional nature of return data. (Even dimensions are optional since a widget can choose to return zero value dimensions.)

Such an api would look like:

// Data contains information about widgets such as dimensions and
overlay content.
// "Data" is deliberately generic referring generally to "widget data". 
// Any other name that communicates this would suffice. 
type Data struct {
    Dimensions
    Overlay op.CallOp
    // ... 
}

// Widget can render itself into a context.
type Widget = func(gtx Context) Data

~eliasnaur 9 months ago

Thank you everyone for working on this.

A viable overlay API should be able to handle an overlay being constrained by the window bounds. The thorniest example I have is a right-click popup menu: it must be oriented to avoid the window edges, and resize itself if the window is too small to contain all menu entries.

Correct me if I'm wrong, but both your suggestions involve widgets preparing a CallOp for the overlay content. However, widgets don't have the window bounds nor its position information to orient and size its popup.

Further, Jack's Data indirection (or additional return values) adds complexity to every Widget function even though very few involve overlays.

It seems to me the right approach is what I think Chris started out with: that overlays are an app-global concern. I glanced at Flutter's Overlay[1], which also seem to be global.

What's missing in Gio to support that approach?

[1] https://api.flutter.dev/flutter/widgets/Overlay-class.html

~whereswaldon 9 months ago

Correct me if I'm wrong, but both your suggestions involve widgets preparing a CallOp for the overlay content. However, widgets don't have the window bounds nor its position information to orient and size its popup.

This is accurate. I wonder whether returning the size and orientation of the overlay might be sufficient though.... Couldn't higher-level render logic then choose to offset the content to ensure that it is still visible?

Further, Jack's Data indirection (or additional return values) adds complexity to every Widget function even though very few involve overlays.

True. Another approach Jack explored was the (infamous) widget interface. Layouts that support overlaying content could test whether a child element implemented an extended widget interface that has an Overlay(gtx C) D method and respond accordingly. This helps avoid modifying the signature of every single widget.

It seems to me the right approach is what I think Chris started out with: that overlays are an app-global concern. I glanced at Flutter's Overlay[1], which also seem to be global. What's missing in Gio to support that approach?

Well, I have it working in materials already ;P. However, it doesn't currently solve all of the problems we need it to solve. It's easy to handle modal drawers and overflow menus because I can infer their "origin" location using heuristics. "This is a left drawer, so I should anchor it to that edge", "this is a top app bar overflow menu in a LTR language, so it should originate from the upper-right corner.", etc... That utterly breaks down when trying to implement something as simple as a drop-down menu (Jack's exact use-case). Gio doesn't currently give us enough information to know where within the frame a specific element is located. Without that, I can't correctly align something like the expanded dropdown options of a select widget to the location of the widget. Jack's current approach elegantly solves this problem by handling the overlay in a highly-local part of the code (the wrapping Flex layout was extended to understand the overlay and to do the right thing).

Right-click context menus are another pathological case. In order to anchor the menu properly, we need to know the exact coordinates of the click. Only the widget that actually receives the event knows those coordinates, and they are relative to its current transform. There isn't a way to properly communicate those coordinates as frame-level coordinates to any overlay logic right now.

So far, I've come up with three approaches (though they are all pretty fuzzy):

  • Use something like Jack's approach so that widgets can return overlay content with dimensional and positional data. Higher-level logic can apply reverse-transforms to the positions and dimensions at each layer to transform the data back into frame-global coordinates, at which time we can resolve the Z-indicies and frame boundaries.
  • Create a second set of layouts that act like Jack's current extension to the Flex layout and advertise support for overlays. Users will be responsible for choosing which layout to use based on their needs. We could treat the layout context within which overlays are possible a little like a monad, and there could be a function that "collapsed" the overlayable content into just normal content (forcing all overlays to be positionally resolved). This would be a function with a signature like func(gtx C, func(C) (D, Overlay)) D, where the overlay is all handled internally to serve as an adapter back to normal gio widgets. This does not address the right-click problem unless you are able to defer resolving the overlay until you know the bounds of the frame, at which point you've just been forced to use these new alternate layouts for your entire UI, and we should just make them the default.
  • Create a new Op that acts as a kind of "beacon." When you want to lay out overlay content, emit a beacon operation with some unique tag and then register the overlay content associated with that beacon. During the actual render pass, resolve the location of the beacon and use it to anchor the modal content. This may be impossible, or stupidly complex. I haven't been deep enough in the render process that we use to understand the scope of change that this represents.
  • As I was re-reading this list, I came up with another option. Is there a way that we can use gio's event system for this? Dispatch an event from a widget that wants to render overlay content... during the render pass, we resolve the absolute coordinates of that widget, then emit it as a system-level event with the absolute coordinates and information about the widget that wanted to make an overlay. It would have a frame of latency between requesting the overlay and actually being able to draw it, but it seems like it would let us resolve the coordinates. Since this idea just popped into my head, it's even-less-fully-baked that the others (and that's saying something).

~jackmordaunt 9 months ago

Thank you everyone for working on this.

A viable overlay API should be able to handle an overlay being constrained by the window bounds.

The thorniest example I have is a right-click popup menu: it must be oriented to avoid the window edges, and resize itself if the window is too small to contain all menu entries.

Correct me if I'm wrong, but both your suggestions involve widgets preparing a CallOp for the overlay content. However, widgets don't have the window bounds nor its position information to orient and size its popup.

Further, Jack's Data indirection (or additional return values) adds complexity to every Widget function even though very few involve overlays.

~eliasnaur

Regarding the multiple returns I agree it adds complexity to the API: not logically, but functionally, since Go will force you to annotate each return type.

However I'm not sure I understand how data indirection adds complexity. It would seem that any widget not using an overlay would leave it alone, and only the specific layouts types would actually care to check for overlay data - thanks to Go's zero values and struct embedding.

The principle at play, as far as I see it, is that the return value of the widget is just "some data about the widget that just rendered itself". It happens to be only dimension data at the moment, but it seems that this is not the only data a widget might want to return about itself.

It seems to me the right approach is what I think Chris started out with: that overlays are an app-global concern. I glanced at Flutter's Overlay[1], which also seem to be global.

What's missing in Gio to support that approach?

[1] https://api.flutter.dev/flutter/widgets/Overlay- class.html

~eliasnaur

Is there a preference to replicate Flutter's layout design choices in Gio?

I did some research on Flutter's approach to Overlays. Basically, in Flutter, you create an "overlay" layer that sits atop everything which has a stack of views (which I essentially just used layout.Flex for). Then further down the widget hierarchy a widget will draw itself. If a widget wants an overlay it will create it and then insert it into the "nearest available overlay". Flutter provides the mechanism to perform such a query.

The overlay management is done imperatively which is the main difference between Flutter's approach and my own.

To reiterate, in Flutter [1]:

  1. Create Overlay widget atop everything else. Navigator includes an Overlay by default.
  2. Widget asks the system for the "nearest overlay".
  3. Widget inserts the overlay widget into the Overlay stack it receives from the system.
  4. Widget has to use funky mechanism for positioning and tracking movement.

Note: this approach hasn't actually solved the context menu problem nor have they come up with a declarative API yet [2].

I wonder whether returning the size and orientation of the overlay might be sufficient though.... Couldn't higher-level render logic then choose to offset the content to ensure that it is still visible?

~whereswaldon

I came to the same conclusion. Rather than returning just an op.CallOp you could return something more rich like:

type Data struct {
    Dimensions // Dimensions of base widget
    Overlay // Optional overlay data
}

type Overlay struct {
	Op op.CallOp // Pre-rendered overlay
    Dimensions // Dimensions of overlay widget
}

Or even just make it a widget closure:

type Data struct {
    Dimensions // Dimensions of base widget
    Overlay // Optional overlay widget
}

type Overlay = func(layout.Context) layout.Dimensions

The difference being that the first one requires the rendering to occur along with the parent widget, and the latter defers rendering of the overlay until it is actually needed.

True. Another approach Jack explored was the (infamous) widget interface. Layouts that support overlaying content could test whether a child element implemented an extended widget interface that has an Overlay(gtx C) D method and respond accordingly. This helps avoid modifying the signature of every single widget.

~whereswaldon

Yes, the reason I moved on from this was because of speculated performance ramifications. Logically it holds that only a widget that implements Overlayable will be considered - letting the API stay very clean. However, it involves an interface type assertion wherever it gets handled. Perhaps this is viable performance wise, depends on profiling I suppose.

The principle between all three approaches (multiple returns, data embedding, interfaces) are the same: the base widget is telling the system that it has some overlay data to draw and the system has some types that check for it, eg the nearest layout.Flex.

This is actually the same principle Flutter, it just uses an imperative "get me the nearest Overlay and then insert my overlay into it" approach.

Use something like Jack's approach so that widgets can return overlay content with dimensional and positional data. Higher-level logic can apply reverse-transforms to the positions and dimensions at each layer to transform the data back into frame-global coordinates, at which time we can resolve the Z-indicies and frame boundaries.

~whereswaldon

I'm leaning toward this. I will take a pass at the context menu problem using this approach.

Regarding "needing the window size", would it be as simple as providing the window bounds via layout.Context and letting the overlay handler, eg layout.Flex, simply bounds check each overlay and apply a transform to keep it inside the window.

[1] https://medium.com/saugo360/https-medium-com-saugo360-flutter-using-overlay-to-display-floating-widgets-2e6d0e8decb9

[2] https://github.com/flutter/flutter/issues/50961

~eliasnaur 9 months ago

Thanks Chris And Jack for the detailed replies. I'm still thinking about the issues, but here's a few comments.

On Tue Oct 13, 2020 at 3:42 AM CEST, ~jackmordaunt wrote:

Thank you everyone for working on this.

A viable overlay API should be able to handle an overlay being constrained by the window bounds.

The thorniest example I have is a right-click popup menu: it must be oriented to avoid the window edges, and resize itself if the window is too small to contain all menu entries.

Correct me if I'm wrong, but both your suggestions involve widgets preparing a CallOp for the overlay content. However, widgets don't have the window bounds nor its position information to orient and size its popup.

Further, Jack's Data indirection (or additional return values) adds complexity to every Widget function even though very few involve overlays.

~eliasnaur

Regarding the multiple returns I agree it adds complexity to the API: not logically, but functionally, since Go will force you to annotate each return type.

However I'm not sure I understand how data indirection adds complexity. It would seem that any widget not using an overlay would leave it alone, and only the specific layouts types would actually care to check for overlay data - thanks to Go's zero values and struct embedding.

By complexitiy I mean the extra fields in Data. It's not much, but somewhat unfortunate every return value needs it.

On the other hand, you could argue Dimensions.Baseline is similarly unfortunate.

It seems to me the right approach is what I think Chris started out with: that overlays are an app-global concern. I glanced at Flutter's Overlay[1], which also seem to be global.

What's missing in Gio to support that approach?

[1] https://api.flutter.dev/flutter/widgets/Overlay- class.html

~eliasnaur

Is there a preference to replicate Flutter's layout design choices in Gio?

No preference. I look at Flutter because that's where Gio's position-less constraints/dimensions layout system is from.

Regarding "needing the window size", would it be as simple as providing the window bounds via layout.Context and letting the overlay handler, eg layout.Flex, simply bounds check each overlay and apply a transform to keep it inside the window.

Yes, but you need both the window dimensions and your own position in relation to the window. Which is not available until after the frame is complete.

~jackmordaunt 9 months ago

I have done some further experimentation with overlays, I'll attempt to articulate what I've learned here.

To render an overlay there are 4 data points you need:

  • overlay layer where the rendering occurs

  • overlay widget to render

  • internal coordinates for the origin of the overlay widget (relative to the overlay container)

  • main axis which to offset by (vertical/horizontal)

Somewhere in the widget tree the overlay needs to be rendered. The overlay location can be an arbitrary distance away from the original widget that the overlay belongs. Typically there'll be a single overlay container at the top of the tree, that spans the dimensions of the window. Considering this, we need a mechanism to get the overlay data from somewhere deep inside the tree, to the overlay container at the top of the tree.

There are two approaches I have come across: injection and bubbling.

#Injection

Flutter uses an "injection" approach. A global reference provided by the context which can be queried to get "the nearest overlay container".

This lets the widget skip passed all intermediate layout containers and give it's data directly to the overlay container.

This method relies on coordinate resolution: Flutter has an API LocalToGlobal that translates local coordinates to global coordinates. This seems to be done by maintaining a tree of "render boxes" which contain coordinate data.

When laying out a widget you will ask for the nearest overlay container, and then inject your overlay content into it, offset by global coordinates.

Since Gio has no retained state between frames, the local-to-global translation would have to be done post-layout. It could perhaps be cached and thus introduce a 1-frame latency. Either way, this coordinate data has to be calculated somehow and must be done external to any given widget.

func (this Widget) Layout(gtx Context) Dimensions {
    dims := this.layout(gtx)
    // overlay is the overlay container closest to this widget in the widget tree.
    // Relies on having some way of traversing the widget tree. 
    overlay = ctx.FindNearestOverlay(this)
    // offset contains the absolute coordinates of this widget's origin.
    // Relies on some retained "render object" being associated with this widget.
    offset := gtx.FindRenderObject(this).LocalToGlobal()
    overlay.insert(OverlayEntry{
    	Offset: Inset{
    		Top: Offset.Y + dims.Y,
    	},
    	Child: this.overlay,
    })
    return dims
}

// layout the widget
func (this Widget) layout(gtx Context) Dimensions {
	//...
}

// layout the overlay
func (this Widget) overlay(gtx Context) Dimensions {
	//...
}

As you can see in the pseudo code, the Flutter approach seems to leverage it's retained nature quite heavily to achieve their API.

#Bubbling

This is the method I chose to implement. It doesn't rely on any retained state, but it does require layout containers to record offsets they made.

Coordinate data is resolved by tracking each offset during layouting. As the overlay data bubbles up, it accrues any offsets that were applied to it's owner on the way down.

Somewhere in the tree is an Overlay layout type. This type inspects the overlay data and renders it offset by summing all the offsets associated with it.

Should the overlay flow out of bounds, it get's transformed to stay in bounds. For example a select input overlay, which typically renders below the select input, would be rendered above the owning widget to keep it in bounds.

If the overlay container should expand to the entire size of the window, simply place the Overlay type somewhere near the top of the layout hierarchy.

#Conclusion

The core problem with rendering overlays appears to be around translating coordinates between contexts. In other words, letting a parent widget calculate where a child widget rendered relative to it's own origin (local to global coordinates being a special case where the parent is at the top of the tree).

In the case of Gio, offset operations are the core of how layout containers are implemented. This means we need some way to track what offsets occur before a widget is rendered. That way the overlay container can position the overlay with the same offsets, aka resolve the internal coordinates.

I have a working sample of the bubbling approach. For now it only considers the y-axis and doesn't resize the overlay to fit constrained spaces.

It is important to note that this approach makes layout containers responsible for tracking the offsets they make so that coordinates can be computed later. There maybe more elegant ways to track offset data.

Fork of Gio here. Fork of materials here to demonstrate the changes.

Run the following and navigate to "Text Field Features".

git clone https://git.sr.ht/~jackmordaunt/materials -b overlays
cd materials && go run example

~whereswaldon 9 months ago

Thanks for your thorough writeup! I think that you correctly identify the necessary information for an overlay system to function, and I appreciate the depth you went into in the approaches that you outlined.

I have some ideas about how to avoid the local=>global coordinate translation using the event system, and I have some questions about the bubbling approach that I need to try to answer via playing with your implementation.

Anyway, I'm posting to say:

  • Thank you for writing this up and implementing a PoC!
  • I'm going to play with your PoC and probably come back with more questions.
  • I'm also going to try to implement a PoC of using the event system to avoid some coordinate translation problems (maybe this idea doesn't work as well as it does in my head; we'll see).

~whereswaldon 9 months ago

Okay, I've reviewed your bubble-up implementation (both the gio and materials components), and I have some thoughts to share:

  • This model definitely works, but it imposes some new constraints upon the widget tree that we need to be cognizant of:
    • Overlay content is collapsed at the first parent Overlay widget. If a widget that wants to draw a global overlay is used within a layout that uses overlays for a different purpose, the nested widget's overlay is collapsed prematurely. This essentially means that no widget can reliably draw at the "top level", as the widget cannot ensure that it isn't being used within an unknown number of nested overlays. To get around this, we could introduce a "level"/z-index construct that would cause an Overlay widget to skip rendering a widget (decrement the "level") and then there could be a special top-level overlay widget that consumed all available overlays regardless of their "level" (though it would need to sort them by it).
    • This makes all new layout/widget code responsible for correctly understanding and transforming the OverlayChild type. If I'm a new gio programmer and I want to make a UI component that can embed an arbitrary widget, I now need to understand how to transform the OverlayChildren of the embedded widget. This will definitely increase the learning curve, though perhaps we could develop utility types to make this easier (if all of the builtin layout types handled it automatically, we could simply caution users to favor those over raw op.TransformOps in order to avoid the problem).
  • This model also may have some problems that it doesn't yet solve:
    • your code can be trivially extended to support the other axis, but in reality I think it needs to support all possible affine transformations, not just offsets. Offsets are by far the most common, but Gio has a really cool story around being able to transform the entire UI without losing functionality (there's a button that spins the whole UI in the kitchen example), and we need to be careful not to lose that feature. Right now, an interface containing one of the Select widgets would be able to be transformed, but the overlaid content would just be statically positioned on the screen.
    • I proposed a model similar to this in a conversation with Elias months ago, and at the time we got stuck on how to properly handle the fact that during layout you are not aware of whether you are inside of a macro or not. If you are, that macro could be transformed an unknown number of times before being actually laid-out. I think your implementation may actually dodge this problem by computing the relevant transformations as you walk up the widget tree (my proposal was computing them on the way down), but I'm not 100% confident. I'd love to hear ~eliasnaur's thoughts on this.
    • I don't know what the performance penalty of this approach looks like. It seems like we'd need to invert every transform at every level on which content was overlaid. The worst-case seems like it could be expensive, but I don't know if that's a show-stopper.

So after reflection, I think that this could be a solid approach if:

  • it was extended to handle arbitrary affine transforms at every level
  • we added some kind of system to control the z-index of overlay content so that deeply-nested widgets could still render overlay content "on top"
  • we found a way to provide a developer experience around this that wasn't full of foot-guns
  • it doesn't carry a crazy performance penalty to use

Thanks again for all of the work and thought that you've put into this! I'm going to work on my PoC of an event-based approach and perhaps we can take the best of both worlds (or maybe my way won't even work; who knows?)!

~whereswaldon 9 months ago

Okay, so I've build a PoC that looks at a different (though related) use-case. I got really obsessed with how to implement a right-click context menu, and I found a way that I think is essentially free (from a performance standpoint), but it still has some design tradeoffs.

Essentially, my approach is predicated on the fact that during event routing, we already know the absolute coordinates of a click event. If we want to precisely anchor a context menu to the location of a click, all we have to do is not discard the absolute coordinates during event processing. However, it is a violation of the Gio abstraction to make logic decisions within an individual widget based on where you are positioned (in absolute space). I got around that by creating a type called Anchor that wraps the coordinates of the click in an unexported form and provides no convenient API to gain access to them. The pointer.Event type gained a field called AbsolutePosition that is the unmodified coordinates of the click represented as an Anchor.

This anchor can be provided to a new layout construct that I called a layout.Overlay (it's just for demonstration purposes, and is less flexible than it could be). If a widget has access to a layout.Overlay, it can request that any layout.Widget be drawn at some Anchor by the layout.Overlay. The widget learns nothing about its absolute position (not violating the abstraction model), but the layout.Overlay is still able to bounds-check the widget to ensure that it doesn't draw off-screen.

You can see my PoC by doing:

git clone https://git.sr.ht/~whereswaldon/gio -b anchor
cd gio/example && go run ./rightclick

It isn't especially pretty, but I tried to capture the classical behaviors of a right-click context menu:

  • when you click outside of it while it's visible, it disappears
  • the item that you click in order to raise it is still clickable independently
  • the contents of the menu are clickable, and there are different options
  • the origin of the menu is the click location unless that would cause the menu to render offscreen, in which case the menu is translated to the nearest onscreen origin that will accommodate it

After building this, I am left with some reservations about it:

  • it doesn't attempt to match the overlay content to the transform of the widget that generated it. For right-click context menu stuff, I don't think that's a big deal. It's usually not perceived as being "part of" the interface in the same way as some other widgets, so having it not be rotated when the rest of the interface wasn't wouldn't be terrible.
  • it doesn't really handle ~jackmordaunt's use-cases like the select dropdown. It just solves the problem of anchoring top-level content to the coordinates of an event.
  • it requires widgets to have a reference to the layout.Overlay in order to construct RightClickAreas. This can be managed with good application design, but it does complicate Gio's usage.

I think the major upside of this approach is that it's basically free computationally (slightly more memory for events, no significant additional computation for transforms). However, we could probably use ~jackmordaunt's walk-up-the-tree reverse transformations to also derive the coordinates of a click in absolute space and could use that to draw context menus in the same way. That approach seems more powerful/extensible.

Another approach that occurred to me was to introduce a new Gio operation that causes an event to be dispatched at the end of the frame. Something like:

type AnchorOp struct {
    LocalCoordinates image.Point
    Tag interface{}
}

During the next frame, you could get the resolved absolute coordinates in a CoordinateEvent or AnchorEvent, and you could provide those to some API for layout like the ones we've been proposing. The advantage of such an implementation would be that we do not need to invert the transforms to obtain the absolute coordinates, though it requires a 1 frame latency to render any overlay content. Not sure how feasible this idea is anyway, as I don't know very much about the processing of the Ops list in the first place.

Anyway, I think I'll stop posting on here and give someone else a chance to share their thoughts.

~jackmordaunt 9 months ago*

I proposed a model similar to this in a conversation with Elias months ago, and at the time we got stuck on how to properly handle the fact that during layout you are not aware of whether you are inside of a macro or not. If you are, that macro could be transformed an unknown number of times before being actually laid-out. I think your implementation may actually dodge this problem by computing the relevant transformations as you walk up the widget tree (my proposal was computing them on the way down), but I'm not 100% confident. I'd love to hear ~eliasnaur's thoughts on this.

~whereswaldon

I'm going to attempt to address this.

#Idea: DeferOp

A defer operation is an operation that gets processed at the end of the frame, after all other operations. When the defer operation is processed, the entire branch of the widget tree (direct path from root to leaf) is accessible for the sake of replaying transforms.

type DeferOp struct {
    ops *Ops // call op?  
}

During the processing of operations you note all stacks / transforms encountered. Once you hit a defer operation, you push it onto a deferred operation list along with transform data.

After the main operation list has been processed, you process the deferred operations and replay the transform data before calling the defer operation.

Thus you have two operation lists one linear, and the other deferred. The deferred list is built up as the linear list is processed. This lets deeply nested widgets defer rendering, which reflects the reality that overlays are just deferred components of a widget, rendered in a different context.

Implementing an overlay would therefore be implemented with a defer operation that records the operations for the overlay itself, and any transforms up to that point in the tree will be replayed for it.

This idea leverages the fact that transformation data is already on the ops list. My bubbling approach is essentially duplicating that data manually in order to replay it (in part because I don't know how to interact with the ops list).

When rendering an overlay in my current approach I'm essentially simulating a "replay" of transforms. (image.Point is a baked form of data; tracking raw op.TransformOp is arguably better and does in fact work).

This pseudo code demonstrates the theoretical processing operations. It is very handwavy but I hope it illustrates what I mean:

// Process the operations.
// `render` is a stub for whatever Gio currently does for handling operations.
func Process(linear Ops) {
	// deferred operations to be processed last.
    var deferred Ops
    // current stack of applicable transforms.
    var transforms Ops

	// track transforms and defers while processing the ops.
    for op in linear {
    	if op == TransformOp {
    		transforms.push(op)
    	}
        if op == DeferOp {
            deferred.push(transforms)
            deferred.push(op)
            continue
        }
        render(op)
    }
    
    // process the deferred operations in linear fashion. 
    for op in deferred {
    	render(op)
    }
}

~whereswaldon 9 months ago

I follow your approach ~jackmordaunt, and it does seem like it would achieve the same thing in a significantly more efficient manner than the previous attempt. However, like you, I have yet to truly grok how the operation list is processed, and I can't really comment on the feasibility of the approach. We'll just have to see what ~eliasnaur thinks when he has time to comment.

~jackmordaunt 9 months ago

Essentially, my approach is predicated on the fact that during event routing, we already know the absolute coordinates of a click event. If we want to precisely anchor a context menu to the location of a click, all we have to do is not discard the absolute coordinates during event processing. However, it is a violation of the Gio abstraction to make logic decisions within an individual widget based on where you are positioned (in absolute space). I got around that by creating a type called Anchor that wraps the coordinates of the click in an unexported form and provides no convenient API to gain access to them. The pointer.Event type gained a field called AbsolutePosition that is the unmodified coordinates of the click represented as an Anchor.

~whereswaldon

Nifty. Unfortunately this seems like it only handles the special case of anchoring to a mouse pointer - which is a specialization of the general problem and, like you said, won't help with select input overlays.

What we need is someway to resolve coordinates of internal widgets. Global coordinates being a special case of a top level container than spans the entire window.

The transform data is the data we need to do this, and it's already in the ops list. I can practically smell the solution!

I will try my had at the DeferOp to see if we can leverage the transform data while processing the ops list.

~whereswaldon 9 months ago

~jackmordaunt: I think we can extend the approach by allowing widgets to emit an Op that anchors a region of their internal dimensions. Perhaps it causes an event to be delivered in the next frame that carries the absolute coordinates of the region (and possibly transform information).

However, your approach looks really promising. I'd rather explore your idea than mine.

Do you have plans for how to handle the case in which a higher-level widget's deferred content overlaps with that of a lower-level one? From my current understanding of the DeferOp proposal, it is impossible for a widget deeper in the widget tree to generate overlay content on top of the overlay content of a parent. Is that correct? Is it desirable? I'm of two minds about whether that is a flaw or not. It seems like a sane limitation, but I just know someone will eventually ask for it.

~jackmordaunt 9 months ago

I think we can extend the approach by allowing widgets to emit an Op that anchors a region of their internal dimensions. Perhaps it causes an event to be delivered in the next frame that carries the absolute coordinates of the region (and possibly transform information).

This would still require someway to resolve the internal coordinates. The 1-frame latency just depends on the strategy used, either wait until the widget tree has been fully computed (the frame), or compute as the stack unwinds (defer).

Maybe I'm misunderstanding :)

Do you have plans for how to handle the case in which a higher-level widget's deferred content overlaps with that of a lower-level one? From my current understanding of the DeferOp proposal, it is impossible for a widget deeper in the widget tree to generate overlay content on top of the overlay content of a parent. Is that correct? Is it desirable? I'm of two minds about whether that is a flaw or not. It seems like a sane limitation, but I just know someone will eventually ask for it.

Yes this is correct. Using a DeferOp as opposed to a return value means that the parent widgets can't access the overlay data of the child directly.

BUT a child could return overlay data in it's API, and let the parent actually perform the DeferOp on it's behalf. For example, the child could simply have an Overlay method. The parent then renders it into a macro, and defers it.

What I think we should avoid, and this is just my intuition so it could be misguided, is the z-index as seen in CSS. The reason being: each widget would have to know (guess) the z-index it should render the overlay at. This creates really tricky data dependencies. This is why in CSS you often see z-index of 999999 just to ensure it sits on top - because the UI programmer doesn't want to juggle each and every z-indexed element.

~jackmordaunt 9 months ago

BUT a child could return overlay data in it's API, and let the parent actually perform the DeferOp on it's behalf. For example, the child could simply have an Overlay method. The parent then renders it into a macro, and defers it.

I take this back. It will fail for the same reason we've being trying to get around: layout containers between the parent and child will hide any transforms they make, which means the parent know where to put the child's overlay.

~eliasnaur 9 months ago

I still haven't thought all this through nor closely read all your posts. So I'm sorry if the below have been discussed:

On Thu Oct 22, 2020 at 10:36, ~jackmordaunt wrote:

What I think we should avoid, and this is just my intuition so it could be misguided, is the z-index as seen in CSS. The reason being: each widget would have to know (guess) the z-index it should render the overlay at. This creates really tricky data dependencies. This is why in CSS you often see z-index of 999999 just to ensure it sits on top - because the UI programmer doesn't want to juggle each and every z-indexed element.

I see the problem with absolute Z values, but it seems to me the layering issue of popups is the easiest to solve with relative, implicit Z values.

In particular, maintain an implicit, unsigned Z value initialized to 0. Then, introduce a LayerOp/ZOp/... that bumps Z with +1. When drawing, sort by Z first, then ops order as usual.

To draw a popup you do something like:

defer op.Push(ops).Pop()
op.LayerOp{}.Add(ops)
... draw popup content ...

nested popups work as expected, and without inter-widget coordination:

defer op.Push(ops).Pop()
op.LayerOp{}.Add(ops)
... draw popup content ...
defer op.Push(ops).Pop()
op.LayerOp{}.Add(ops)
... draw secondary popup ...

With LayerOp/ZOp I don't see a need to know global coordinates for right-click popups and dropdowns; you draw the popup in local coordinates, and the LayerOp makes sure it appears on top of everything else.

Another thing that occurred to me is, are we spending too much effort making sure popups can resize and position themselves inside their owner window? In other toolkits, popups are not clipped to the window bounds and therefore it isn't necessary to be clever with position and size. Of course, implementing Gio support for drawing outside the window is not trivial, and there is many advantages to keeping all UI inside the window, in particular for fullscreen windows and on mobile.

Note that I don't have a good answer for the case of a modal centered in its window, expanded to take up all available space.

Elias

~jackmordaunt 9 months ago*

Another thing that occurred to me is, are we spending too much effort making sure popups can resize and position themselves inside their owner window?

I was basing my behavior on browser implementations. For example if you look at material.io, try to resize the window very small.

The overlay responds in 3 ways:

  1. Flips to the side with the most space.
  2. Resizes itself.
  3. Renders atop even the nav bar.

While this exact behavior is not strictly necessary, it represents what many users are going to expect since they've been trained on browser UIs.

In terms of interacting overlays: this never even occurs because only one select is "active" at any given time. z-index is still valuable in other contexts, but here it simply gets avoided.

~jackmordaunt 9 months ago

defer op.Push(ops).Pop()
op.LayerOp{}.Add(ops)
... draw popup content ...
defer op.Push(ops).Pop()
op.LayerOp{}.Add(ops)
... draw secondary popup ...

I like this API. I'm willing to drop the resizing requirement! To me it's a nicety to have parity with the browser but I'm not tied to that.

~eliasnaur 6 months ago

I've pushed a variant of my LayerOp idea, the op.Defer function. Take a look at gioui.org/commit/f86703e and let me know whether it solves the layering issues of modal boxes.

~eliasnaur 6 months ago

Chris Waldon et al pointed out that restoring all state is not always preferable; in particular, the clip state is unwanted for a popup. https//gioui.org/commit/d688673 changes Defer to only restore the transformation.

~whereswaldon referenced this from #195 5 months ago

~inkeliz 5 months ago*

I'm playing the op.Defer for while, I already notice some problems while using it. Consider the following code (https://play.golang.org/p/Lq2Kk6ZL7-J):

	// First Macro - Black
	op.Defer(gtx.Ops, Macro(gtx, func(gtx layout.Context) {
		clip.Rect{Max: gtx.Constraints.Max}.Add(gtx.Ops)
		paint.Fill(gtx.Ops, color.NRGBA{0, 0, 0, 255})
	}))

	// Second Macro - Red
	op.Defer(gtx.Ops, Macro(gtx, func(gtx layout.Context) {
		clip.Rect{Max: gtx.Constraints.Max}.Add(gtx.Ops)
		paint.Fill(gtx.Ops, color.NRGBA{255, 0, 0, 255})

		// Inner Macro - Blue
		op.Defer(gtx.Ops, Macro(gtx, func(gtx layout.Context) {
			clip.Rect{Max: gtx.Constraints.Max}.Add(gtx.Ops)
			paint.Fill(gtx.Ops, color.NRGBA{0, 0, 255, 255})
		}))
	}))

	// Third Macro - Green
	op.Defer(gtx.Ops, Macro(gtx, func(gtx layout.Context) {
		clip.Rect{Max: gtx.Constraints.Max}.Add(gtx.Ops)
		paint.Fill(gtx.Ops, color.NRGBA{0, 255, 0, 255})
	}))

This code have multiples op.Defer, all of them changes the color of the "screen". I think that the result would be "Green", since it's the last op.Defer in the code. However, the result is "Blue", which is the op.Defer inside another op.Defer.

That behavior is really dificult to handle. The only way to fix that is doing:

	op.Defer(gtx.Ops, Macro(gtx, func(gtx layout.Context) {
		op.Defer(gtx.Ops, Macro(gtx, func(gtx layout.Context) {
			op.Defer(gtx.Ops, Macro(gtx, func(gtx layout.Context) {
				clip.Rect{Max: gtx.Constraints.Max}.Add(gtx.Ops)
				paint.Fill(gtx.Ops, color.NRGBA{0, 255, 0, 255})
			}))
		}))
	}))

That code will enforce that the "Green" have a "deeper-op.Defer" than the "Blue". In the end: for every nested op.Defer I need to create a new op.Defer to make the last-one longer than any other.

~eliasnaur 5 months ago

On Thu Feb 25, 2021 at 18:43 CET, ~inkeliz wrote:

I'm playing the op.Defer for while, I already notice some problems while using it. Consider the following code (https://play.golang.org/p/Lq2Kk6ZL7-J):

	// First Macro - Black

	op.Defer(gtx.Ops, Macro(gtx, func(gtx layout.Context) {
		
clip.Rect{Max: gtx.Constraints.Max}.Add(gtx.Ops)
		paint.Fill(gtx.Ops, 
color.NRGBA{0, 0, 0, 255})
	}))

	// Second Macro - Red
	
op.Defer(gtx.Ops, Macro(gtx, func(gtx layout.Context) {
		
clip.Rect{Max: gtx.Constraints.Max}.Add(gtx.Ops)
		paint.Fill(gtx.Ops, 
color.NRGBA{255, 0, 0, 255})

		// Inner Macro - Blue
		
op.Defer(gtx.Ops, Macro(gtx, func(gtx layout.Context) {
			
clip.Rect{Max: gtx.Constraints.Max}.Add(gtx.Ops)
			paint.Fill(gtx.Ops,
 color.NRGBA{0, 0, 255, 255})
		}))
	}))

	// Third Macro - Green
	
op.Defer(gtx.Ops, Macro(gtx, func(gtx layout.Context) {
		
clip.Rect{Max: gtx.Constraints.Max}.Add(gtx.Ops)
		paint.Fill(gtx.Ops, 
color.NRGBA{0, 255, 0, 255})
	}))

This code have multiples op.Defer, all of them changes the color of the "screen". What I suppose is that the screen will be "Green", since it's the last op.Defer in the code. However, the result is "Blue", which is the op.Defer inside another op.Defer.

That behavior is really dificult to handle. The only way to fix that is doing:

	
op.Defer(gtx.Ops, Macro(gtx, func(gtx layout.Context) {
		
op.Defer(gtx.Ops, Macro(gtx, func(gtx layout.Context) {
			
op.Defer(gtx.Ops, Macro(gtx, func(gtx layout.Context) {
				
clip.Rect{Max: gtx.Constraints.Max}.Add(gtx.Ops)
				
paint.Fill(gtx.Ops, color.NRGBA{0, 255, 0, 255})
			}))
		}))
	}))

In the end: for every nested op.Defer I need to create a new op.Defer to make the last-one longer than any other.

This behaviour is expected, in that the Blue defer is added inside a defer and therefore run last.

Can you sketch a use-case where you need the ordering you mention? I think we may be able to change op.Defer to run nested defers immediately after their containing defers.

Elias

~inkeliz 5 months ago*

Can you sketch a use-case where you need the ordering you mention? I think we may be able to change op.Defer to run nested defers immediately after their containing defers.

In my use case, my design is the following:

  • Header
  • Body

The Body almost always use some layout.List to be able to scroll. I want to "Header" be always visible, so I use the op.Defer for Header and I render it after everything (using op.Offset and op.Stack) to enforce the order. So, in order of rendering it is: 1. Body; 2. Header.

If the "Body" contains some "Modal", it works. The "Modal"/"Pop-Up" will overlay any "Body" content, but doesn't overlay the "Header". That is what I expect and what I get. Fine.

However. One of my "Editor" have another op.Defer. So, if I put one "Editor" inside the "Modal" of the "Body" (in the end: Body -> Modal -> Editor), it breaks everything. The Editor can overlay the Header, since it's one op.Defer inside another (the modal).

The only way to fix that, is change the Header to 2x (or 3x) op.Defer. I'll try to make some in-code example of it.

~eliasnaur 4 months ago

Thank you for elaborating. I can kind-of-sort-of see an issue, but I'm not sure. It would be very helpful to have a simple example with Header, Body, Modal, Editor as, say, colored rectangles.

The reason I ask for more detail is that the issue you're seeing is not a bug, but rather a consequence of how op.Defer is defined ("run after all other operations have completed"). So I'd like to have a compelling case before changing that definition.

~stalker_loki 2 months ago*

Apologies in advance if I am just repeating anyone else's idea, as I have skimmed, I think that the defer op idea is a good one, and it can solve more problems than zindex for overlays.

The defer op would function as a 'last op only' each new push to the list would become the one that functions, as in the drawing order, the one at the top is rendered last, and thus its' defers are the ones that should be processed, and the others discarded. This would be done by using a loop to drain the channel into a predeclared variable and then processing it at the end of the render queue.

This is something that could do more than just determining the topmost, active layer, it can also deal with any other type of concurrent requests that should only operate the last one and discard the rest. Such as identifying which tab of a stack selector is active, since its operations also have the same property as the zindex problem - its ops are the ones that get to the GPU, and thus its deferred operations end up at the end of the queue.

The atomic FIFO design of CSP channels enables this kind of programming, which stays more within the idiom of Gio itself, as well as exploiting the language's native use of it.

Oh. yes, of course, this isn't a pure channel based design problem, the queue is, however FIFO and the same principles apply.

I of course prefer to move this into the run loop, as it enables the direct use of channel syntax. That's probably not relevant as you can wrap channel sends with functions anyway.

~gedw99 a month ago

I see in this conversation a reference to how flutter does overlay.

The way flutter does it is not that well received in the flutter community and this flutter lib is quite popular

https://github.com/rrousselGit/flutter_portal

It embraces “ everything is a widget” approach and does not force overlays to be a special case .

~gedw99 a month ago

I see in this conversation a reference to how flutter does overlay.

The way flutter does it is not that well received in the flutter community and this flutter lib is quite popular

https://github.com/rrousselGit/flutter_portal

It embraces “ everything is a widget” approach and does not force overlays to be a special case .

Register here or Log in to comment, or comment via email.