Skip to content

Generalize the type of keepAlive#, touch# #152

Closed

Description

@bgamari

Up until now the types of keepAlive# and touch# have precluded their use in ST:

touch#
    :: forall (lev :: Levity) (v :: TYPE (BoxedRep lev)).
       v -> State# RealWorld -> State# RealWorld

keepAlive#
    :: forall (lev :: Levity) (v :: TYPE (BoxedRep lev))
              (rep :: RuntimeRep) (p :: TYPE rep).
       v -> State# RealWorld -> (State# RealWorld -> p) -> p

In GHC #23163 it was proposed that we generalize this:

touch#
    :: forall (lev :: Levity) (v :: TYPE (BoxedRep lev)) s.
       v -> State# s -> State# s

keepAlive#
    :: forall (lev :: Levity) (v :: TYPE (BoxedRep lev))
              (rep :: RuntimeRep) (p :: TYPE rep) s.
       v -> State# s -> (State# s -> p) -> p

As this is re-exported from GHC.Exts in base, we believe this may require a CLC proposal

Activity

  1. changed the title [-]Generalize the type of keepAlive#[/-] [+]Generalize the type of keepAlive#, touch#[/+] on Mar 28, 2023
  2. treeowl commented on Mar 28, 2023

    @treeowl

    Yes, please!

  3. chessai commented on Mar 28, 2023

    @chessai
    Member

    This would be very useful

  4. phadej commented on Mar 28, 2023

    @phadej

    This feels silly that this change needs CLC proposal, but apparently doesn't need GHC proposal... (after all, this is changing GHC primops!)

  5. chshersh commented on Mar 30, 2023

    @chshersh
    Member

    I have no idea what those two functions do.

    • keepAlive# docs don't really help
    • touch# doesn't have documentation at all

    I think this is the case of something that is supposed to be internal but somehow got leaked via reexports from ghc-prim to base.

    Even though these two functions look like a really internal thing, I see that they have usages across multiple popular packages:

    This looks like a change that could break things. So impact assessment is required.

  6. added
    awaits-impact-assessmentAwaits an impact assessment (e. g., using https://github.com/Bodigrim/clc-stackage)
    on Mar 30, 2023
  7. konsumlamm commented on Mar 30, 2023

    @konsumlamm
    Contributor

    I think this is the case of something that is supposed to be internal but somehow got leaked via reexports from ghc-prim to base.

    Even though these two functions look like a really internal thing, I see that they have usages across multiple popular packages:

    They are not supposed to be internal, they are primops. They are used e.g. in primitive (which is where this issue comes from in the first place, see haskell/primitive#375).

  8. chshersh commented on Mar 30, 2023

    @chshersh
    Member

    They are not supposed to be internal, they are primops.

    In my vocabulary, primops and internal are the same thing: something that is not supposed to be used by a median language user, and instead should be used only for some low-level stuff (like primitive).

  9. bgamari commented on Mar 30, 2023

    @bgamari
    Author

    @chshersh this blog post may shed some light on the intended usage of these operations.

    For what it's worth, I generally agree with @chshersh that primops aren't necessarily intended to be widely consumed. There are many primops which are present only to implement bits of GHC (e.g. unpackClosure#), others which are intended to be exposed only via their friendlier lifted wrappers (e.g. indexInt16OffAddr#), and others that are hard enough to use correctly that a user should expect to review their usage with every new GHC release (e.g. addrToAny#).

    That being said, I think keepAlive# and touch# are examples of primops that do have legitimate uses in low-level library code. However, I cannot see any way in which this generalization would break an existing well-typed program. The only way that such a generalization would break is a failure of type inference; however, all uses of keepAlive# and touch# are already specialised to RealWorld.

  10. chshersh commented on Mar 30, 2023

    @chshersh
    Member

    @bgamari I see. Thanks for the detailed response! I can accept this as an impact assessment and let everyone else to express their concerns and suggestions.

  11. removed
    awaits-impact-assessmentAwaits an impact assessment (e. g., using https://github.com/Bodigrim/clc-stackage)
    on Mar 30, 2023
  12. Bodigrim commented on Apr 2, 2023

    @Bodigrim
    Contributor

    I think this is the case of something that is supposed to be internal but somehow got leaked via reexports from ghc-prim to base.

    FWIW entire ghc-prim is re-exported from base, usually from GHC.Exts, that's a proper operation.

    The idea is that users should never import ghc-prim themselves: this package is deeply magical in its structure and can (and does) change rapidly without any deprecation period or warning. Using its entities through base:GHC.Exts allows us (at least in theory) to provide some compatibility shims and wrappers.

    I have no idea what those two functions do.

    * `keepAlive#` docs don't really help
    
    * `touch#` doesn't have documentation at all
    

    I must admit that this is a stumbling block for me as well, despite a non-zero experience in low-level guts of Haskell. It would be immensely helpful, if someone volunteered to improve the documentation.

    @bgamari @andrewthad @chessai @treeowl is there a good reason to touch touch#? My understanding of https://www.haskell.org/ghc/blog/20210607-the-keepAlive-story.html is that all new development should use keepAlive#.

  13. cartazio commented on Apr 3, 2023

    @cartazio

    For those who aren’t familiar with them: both operations are to ensure some data structure stays reachable on the heap.

    As for why touch still exists: there exists code out there that largely uses touch correctly in a way that won’t be miscompiled, and there’s no good reason to break it. Otoh its use for new code should absolutely be discouraged.

  14. treeowl commented on Apr 3, 2023

    @treeowl

    @cartazio there are also situations where keepAlive# has performance issues, or (at least potentially) where object lifetimes are supposed to be staggered.

  15. cartazio commented on Apr 3, 2023

    @cartazio

    Totally. Point being keeping both has very real value and low marginal cost.

  16. Bodigrim commented on Apr 3, 2023

    @Bodigrim
    Contributor

    @cartazio There've been no suggestions to remove touch#.

    @treeowl could you please elaborate on performance issues with keepAlive# with a specific example?

  17. treeowl commented on Apr 3, 2023

    @treeowl

    @Bodigrim See Lost optimisation: CPR unboxing in compiler/GHC/CoreToStg/Prep.hs. I may have been wrong about the staggering bit; sorry.

  18. bgamari commented on Apr 5, 2023

    @bgamari
    Author

    @bgamari @andrewthad @chessai @treeowl is there a good reason to touch touch#? My understanding of https://www.haskell.org/ghc/blog/20210607-the-keepAlive-story.html is that all new development should use keepAlive#.

    While we do strongly recommend that people use keepAlive# whenever possible, sometimes this is simply not the case. Specifically, some programs use ByteArray#s which whose lifetime does not follow the lexical scoping of the program. In these cases one must use touch# (and be extremely careful while doing so).

    While the original ticket only addressed keepAlive#, I would argue that it makes sense to treat these operations symmetrically.

  19. Kleidukos commented on Apr 5, 2023

    @Kleidukos
    Member

    (@bgamari you don't have to answer right away, but I have a question: Do these programs use ByteaArray#s in such a way because there is no other way, or could this usage be modified as not to require touch#?)

  20. bgamari commented on Apr 5, 2023

    @bgamari
    Author

    (@bgamari you don't have to answer right away, but I have a question: Do these programs use ByteaArray#s in such a way because there is no other way, or could this usage be modified as not to require touch#?)

    I can certainly envision cases where the structure of a foreign library essentially precludes lexical scoping of a buffer.

  21. Bodigrim commented on Apr 14, 2023

    @Bodigrim
    Contributor

    Thanks for the explanations @bgamari.

    Dear CLC members, any non-binding opinions on this?

  22. Bodigrim commented on Apr 17, 2023

    @Bodigrim
    Contributor

    Dear CLC members, given that there are no more comments, let's vote on the proposal to generalize keepAlive# and touch# to accept any state token, not only RealWorld, as detailed in https://gitlab.haskell.org/ghc/ghc/-/merge_requests/10171/diffs. See https://gitlab.haskell.org/ghc/ghc/-/issues/23163 and haskell/primitive#375 for related discussion.

    @tomjaguarpaw @chshersh @mixphix @hasufell @parsonsmatt @angerman


    +1 from me.

  23. mixphix commented on Apr 17, 2023

    @mixphix
    Contributor

    I'm in favour!

  24. chshersh commented on Apr 17, 2023

    @chshersh
    Member

    +1 from me

  25. parsonsmatt commented on Apr 18, 2023

    @parsonsmatt

    +1

  26. Bodigrim commented on Apr 19, 2023

    @Bodigrim
    Contributor

    Thanks all, 4 votes are enough to approve.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    approvedApproved by CLC votebase-4.19Implemented in base-4.19 (GHC 9.8)

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions