This is my notebook (later renamed Journal so we can launch it with cmd-j). I started it during a session of Hypertext 22†. This is intended to be a completely regular Author document with only two features added in Author to make it a Notebook, (quick access to open the document and quick to enter data and location as described further below). The use case is a notebook for the user to add to for years and years and the Author functions helps the user find things later. An issue is that users can add to do items but forget to do them. That is why this allows for a simple way to designate something as a to do and then remove that designation later, and maybe more importantly, have it shown to them on normal fold even if they might not be looking for it. This is why there is an easy way to add date stamps as well, to help the user find in time, for any items then felt the value of adding time to.
My pitch for the Notebook in Author:
Quick-click in Author to open your Notebook. This is simply an Author document with all the functions of Author to find and interact with. That’s the key, it’s simple
Two specific features are available to make the notebook more useful:
- Click to insert the date, should you wish, which appears when you fold the document into an outline to help you find information in the future.
- Click to turn a note into a To Do item which appears in red when you fold into an outline to remind you that you have not done it.
- Quick opening.
- Ideal open: Click on ‘Notebook’ at top level.
- Second best: Click on ‘Notebook’ as shown in the attached screenshot.
- [Keyboard shortcut cmd-§ or cmd-‘ (depending on keyboard layout, with § for non-US keyboard) or ctrl-n or ctrl-shift-n]
- Date for context. It needs to have a keyboard shortcut for date and location entry as text. Ctrl-d or ctrl-shift-d Or ideally in the ctrl-click menu on not selected text. Maybe make this level heading or highlight or something else to see in fold? Maybe by formatting? I think as indicated above, which the user can of course mess with. Simply list the date and location, but date highlighted.
- To Do list support. If [] is added to the start of a sentence, empty, it makes it a To Do item. In fold, the text on the same line following it is red (though not the []).
- Highlighting the key text in each paragraph in a document.
- Text analysis to help the author remove unnecessary words and properly check grammar.
- In a folded view showing only headings, summarise the body text so that each section has a summary.
- Colour words based on categories, as I played around with Doug, so that technology words are yellow, people’s names blue, company names red and so on, for example, to help skim.
- Feed an AI all your previous work, maybe in Author, automatically, then have a keyboard shortcut to allow the AI to suggest the next sentence.
- Use ML/AI (the terms seem to be a bit mixed these days) to analyse your document to see if there are contradictions.
- Check for style. In my case my PhD thesis is suffering from too personal style, and I need it edited to be more academic, more neutral.
- And a major one I thought of today. Imagine if you can set a category for different books, which are accessible via Google Books perhaps, for how you see them politically. You can say that this book is supportive of the democratic viewpoint, this other one is authoritarian and so on. These are then analysed so that they can be compared to any new text you throw at it, not only to see which type it is but also to write a summary from all the perspectives! This means that you will be presented by one summary which is from the perspective that corporations are most important, another from the perspective that democracy is most important and so on. I have said a few times recently that one of the most profound things a person can do is choose where to stand, where to see from, and this could help.
- Another big one: Take all the transcribed text from a series of meetings which are ongoing, as many companies have and we have twice a week, and analyse them and add a summary as well as showing if something changed over time.
- Enter YouTube URL of recording.
- Specify time of day the recording started.
- Button to click to turn into plain text Date and time.
That’s it, what do you think?
Hypertext 22
30 June 2022 10:42 Four Points by Sheraton Barcelona Diagonal Hotel, Avinguda Diagonal, 161-163 08018 Barcelona, Spain
Sam Brooker presents at Hypertext ’22, my notes:
Different kinds of intention for fiction or academic?
Is an author different in fiction and factual work?
Notebook for Author?
I need a notebook/ feature in Author. Just a normal doc but cannot be deleted and has easy access.
To build:
Opens a document called ‘My Notebook’ or creates a document called My Notebook, in Documents/Author.
Also allow user to select text and choose, at the bottom of the ctrl-menu: ‘Make To Do’ which simply adds ‘[]’ before the text. User put a glyph inside, such as • or √ or anything and it no longer shows as to do in red in fold.
Discussion in Hypertext 22 open discussion
People don’t ‘get’ the hypertext term. Maybe meta text? Or maybe focus on the connector? Not necessarily text, hypermedia.
Also, where are the other people from the conference?
Claus says maybe manifesto to define hypertext.
Naming: Cybertext? Dene refers to cyber mountain
Hypertext as tools for thought, hypertext as conceptual tool for thought
Writing for a specific medium for reading was before, Mark said once that we now write for reading which may be different than we are aware of. Hypertext eduction thus can be about the importance of learning to read and author connected writing maybe?
Mapping what is known and not known in hypertext, mention by Sam. I’d say addressability is an open issue. I am writing these notes here, talking in the group, which will be on YouTube at some point, and pasting some of my thoughts in the group slack channel. How does it connect? Why do I have to write and paste elsewhere? Why can I know transclude or at least tunnel?
Dene asks if we can have a best practice for hybrid meetings, including such things as cameras on (esp in small group meetings) and keynote speaker physically present etc.
I’ll add we need new tools for textual groups!!! Research project. Not just faffing about in Slack and Zoom and Twitter and more. We can do better right? Live a deeper, more hyper textually connected literacy.
Is hypertext a deeper literacy? Discussion on the tapback.
Richer interactions for conference proceedings please. Of course I think visual-meta can help with this. https://visual-meta.info
Can we make our proceedings so richly interactive that they really illustrate what hypertext *is*, without having the reader need special software?
I feel we gotta work to make hypertext better. Live it.
(Where do these discussions ‘go’ once we have stopped talking?…)
‘More Comfortable with Chaos’ talk. Came in late and spent 5 mins setting up.
Level of complexity Annie Murphy Paul discusses in The Extended Mind. Edgar friends annoying: don’t listen.
Catherine Haylee, how we think, mentioned by Dene (I can’t find).
Mark Anderson & Dave Millard presentation
https://youtu.be/7VzJlC6K06k video I took of him after he credited our Future of Text group.
Adam Wern, the question Brandel made and with visual-meta is pretty auto right? Guy brought up faceted browsing, which might also be faceted search.
[] If Adam can make an API/URL scheme, then maybe in Author/Reader we can give the user the option to open the citation in this view, if available.
Not keyword if no lock. Better metadata of actual contents.
Thoughts on moral analysis presentations
Original
Virtue – Vice
Authority – Subversion
Care – harm
Fairness – cheating
Loyalty – betrayal
Purity – degradation
Maybe change?
Status quo – Change
Authority- Subversion
Care – harm
Loyalty – betrayal
Purity – mixing
Maybe remove?
Fairness – cheating
Virtue – Vice
Friday.
Daniel Rossner, Claus Atzenbeck, Tom Gross.
Less interactions in a spatial hypertext system which is great since more is available at a glance. An example of interactions being in the way.
Did you somehow measure the impact of the suggestions which appeared?
How is funding? Maybe we make it best Spatial Hypertext paper of Hypertext 22 to help with profile?
Magdocs. Very cool, should tell Vint to tell to Google Docs team
Change of Notebook to ‘Journal’
Cmd-j and no To Do, simply make text red and show in folded view
Dene Keynote
Informs the future. Contains valuable mental travels.
Maybe guided videos by pro gamers/streamers / artists / readers to add?
Great to see the caged visual and all the experimental work!!!!!
Frode Blue Sky
Mention also about the long term. Not like Dene rich presentation inherently, but provide potential for rich interaction.
Back to London
And now I’m on my way back to London with Mark Anderson on a most unpleasant flight, after a 2 hour delay. I’ll try to remember to paste in what I texted Emily here when I land. My very small amount of unpleasantness does really highlight how terrible it is in Ukraine, where so many as so harassed and much, much worse. Anyway, it was unpleasant and the drive home was not all that much better. Quite a fast and aggressive driver but he seemed young and in charge. He was south asian/indian and it was weird because my two closest friends, Harsha and Enamul also were but in this context with him being a driver it was such a separation and I almost wanted to say something, which would have been quite pathetic. It’s very sad, heavy and also odd to have such a large part of my life and identity gone. I want to talk to Janine about this, she is back from some Spanish island today, having been there to spend some time celebrating her mum’s birthday. She is going through quite a tough time but she has grown and not only is she resilient, she is also aware of those around her. She will be all right.
On return to London I aim to make a few user guide videos of Author for marketing and complete the two issues of the Journal. I think I may need to also start using auto transcription since the human ones are getting expensive and slow and still require quite a lot of editing.
Edgar woke up to his new owl this morning, and he says it speaks Spanish, after telling me quite a few things about arctic owls. I just love how he is learning and connecting. When we went to the station this morning (him and mummy for Japanese in Sloane Square, me to pick up the Tesla at Heathrow), I asked him how much I missed him and he said very very very and then I asked how much he missed me and he said thousand very. I then told him it was important to my work and it was very interesting with these people and he asked if I was ‘curious in Spain’ and goodness how wonderful it was to hear him speak like that.
So now I am on the train to Heathrow, having written the few paragraphs above, except the one which starts ‘On return to London…’ which I wrote on the plane. I am comfortable with this Journal working like that, just like I might very well have added notes before and after previously written text in a paper Journal.
2 July 2022
Further Journal Thoughts
I have decided on the name ‘Journal’ so we can use cmd-j to launch it and changed the two new capabilities (red and date) to be much less important, since highlight in white is also ok and date can be typed. I hope Jacob can implement this, fix the Author bugs and simplify the Define Concept dialog this month for release for students.
And I need to make one walkthrough demo every day now, release on twitter and YouTube then later on TikTok and Instagram.
(Update 7 July). I now have cmd-J to load it, but noting else, and I’ll test for a few weeks and see.
Bob Stein on FoT 4 July 2022
He says using the tapestry with elements in the same visual field is “Way more different” than expected.
Vint typed “this is context preserving”.
I’ll mention Bob Horn Mural, then refer to different and context for VR use…
So that made sense.
Regarding this has been a dream for 40 years.. Why develop? Not only what is possible, also commercial pressure or augmentation opportunity shapes
Journal discussion: General tools vs yet another button. This is why Journal in Author, not separate app.
Definition Dialog Bold improvement
Mark Anderson on Definition dialog which now shows terms which are in the definition in bold, so it can be clicked on, and he makes the remark that maybe a back button could be useful. (Also, Boris Johnson just resigned). Mark didn’t mind the bold vs underlined and agrees that underline would too much signal link out of the application to the browser.
Words are interesting write
Notes about the recording.
Why VR? (Draft)
As humanity dons their VR/AR headsets over the next few years (as we surely will, with Meta releasing updates to its hugely successful Quest and Apple releasing their own, with an expected high level of consumer friendly polish), we will enter cyberspace to a degree our laptops and smartphones have previously only hinted at.
A new world
This all-encompassing digital experience, where everything in our vision will be mediated by computer systems, as well as everything we hear and later every physical sensation, will be a huge leap for mankind.
Dreaming
We owe it to our future selves and our descendants to look at this new landscape and decide what it should be–after all, this new world will be entirely human made. This cannot be made only by those mega corporations which are investing billions in VR,
I believe that the most powerful human capability may very well be our capability to dream.
A professor, whose name I cannot remember, from Syracuse University once said that “truth kills creativity”. What he meant by this is that once something is established to be something, that becomes a truth and then not questioned. An important and relevant example is how the personal computer became a rectangular box with a pointing device and keyboard, running an operating system with applications for writing, reading, browsing the web, working on spreadsheets and doing email in the 1990s, with much of the potential explored by Doug Engelbart, Ted Nelson and other pioneers in the 1960s no longer considered to be “what a computer is”. What a computer ‘is’ continues to develop, but only very slowly, with a pandemic being necessary for video conferencing to finally take off.
We no longer dream what a personal computer can be. We believe we know what it is.
We no longer dream how we can interact with text on a computer. We believe we know how it can be done. This is the reason I have hosted the Future of Text Symposium for all these years-to help us revisit and think anew. But text interactions on digital systems exist, so it is very hard to un-learn and re-dream.
This is not the case with VR. We still have the mental space to dream of what it can be. And I think it is vital that we do so, before VR is implemented and we ‘know’ what VR ‘truly’ is. We cannot afford to be complacent. We cannot afford to let this opportunity go to waste. We cannot afford for VR to be predominantly developed for the enrichment of the large companies which produce the hardware and operating systems.
Issues
Issues with the development of VR which are already apparent include such basic questions as:
How we will be able to navigate in VR from one ‘room’ to another. Picture yourself at Hampton Court Place at the time of Henry VIII and after enjoying the experience and learning something about this time, you wonder, quite randomly, how Hampton Court as used during World War II. How can you now find out if there is any available VR data for that location at that time? Will we need to download an app to enter this time? If so, how will we find and evaluate this app, will we need to exit the current room and go through menus? How will the provider of the other rooms be presented, will it be made by companies we can choose how we feel about? Would we prefer a BBC, History Channel or Fox News view of history? Or will we trust what a small startup has made?
This leads to deeper issues of how data will connect. Will there be protocols for searching along timelines and geographies? How can data connect for fact checking?
What about creating knowledge products in a room made by one vendor and then having a meeting in a room made by another–will the knowledge product be the same? Will you be able to bring it with you or will we have compatibility issues as we had with Mac and Windows, the browser wars and Microsoft Office and Mac? If so, this will be a compatibility issue of all-encompassing proportions.
A dream
VR has the potential to further alienate us by allowing us to dive deeper into filter bubbles where we only see what we approve and our interactions become increasingly virtual and distant. We believe that VR also has the potential to connect us to each other and our knowledge deeper than we have previously imagined, truly augmenting who we are to tackle the problems we face today and in the future.
VR can impression us or it can deeply connect us and empower us. It is our collective choice as to how we street the development of this new medium, this new world.
Work
If we don’t all work on this, we cannot expect VR to work for all of us.
truth kills creativity
Tom Lombardo “truth kills creativity” https://twitter.com/hervadasala/status/1546382385687805952?s=20&t=EFBfrJdSp6Stdv8gLA6dHQ Francesc Hervada-Sala found it for me at https://t.co/m9pYRGt5MK
Future of Text 2022 Who to invite
Email to group: And please also tell me who you would like to invite. I think artists, academics and technologists as well as theoreticians should be there. Karen Armstrong’s new book ’Sacred Nature’ (highly recommended) talks about how we need to change our spirituality back to a reverence for all of nature, not just to some distant/higher ‘god’, in order to have our priorities straight for dealing with our planet and each other. I think she is right and that this is an existential issue. Therefore a deep question about VR, I think , will be if VR will alienate us or bring us closer to each other and the world. We need to look at that, not only the technical side of course.
2022 Symposium Website 12 July 2022
We look at the future of text because we feel it is important.
(Why Text?) https://thefutureoftext.org/2022/07/11/why-text/
Now that VR is about to reach mass usage, we look at text in VR, specifically for knowledge work.
(Why VR?) https://thefutureoftext.org/why-vr/
VR & Knowledge Work
This symposium looks at the future of text in VR from the perspective of knowledge work. It seeks to explore practices, policies, and possibilities that present are now and are laying ahead so that those working in this area of scholarship can lend their voices to the ongoing development of VR technologies and find effective ways to incorporate them into our work.
To clarify, this is about knowledge work in VR outside the clearly mapped 3D systems such as CAD and outside the social side of work, as well as games and entertainment in general, which are receiving investment already. Further, we do not expect VR to be the exclusive medium through which we interact with text, but rather that we will interact with text in VR alongside traditional digital as well as analog media in a ubiquitous computing environment.
We Ask
What will text be when expressed in VR environments—when words are evoked through touch, interacted with through bodily movements, and is immersed with us in 3D space? How can working with text in VR augment how we think and communicate?
Since entering VR will be a much more personal experience than we are used to through flat screens, we ask how will VR change us and how might we need to change to flourish in VR rather than disappear in VR. What will it mean to be human when we are fully immersed in a digital environment? Can we build VR to connect us closer to each other and the natural world or are we bound to use VR to further isolate ourselves?
What will it be like for children of the future to grow up in worlds with no distance and with infinite possibilities? Will their reach be extended or will they lose perspective?
In other words, how might VR be developed to bring out the best in us?
Why You?
A better future will not be automatic. Developments which only a few decades ago would have seemed like magic, or at least like science fiction are just around the corner, the results of massive investment by large companies.
We try to look at what aspect of work in VR can not be taken for granted because they can not be expected to be developed by the commercial developers of VR systems since they will not directly benefit the cashflow of these companies.
The needs of knowledge workers do not perfectly overlap those of the companies producing the VR experiences.
Why Now?
We think this is particularly important to do this work before VR environments become commonplace and whatever the giants of VR decide will be the way it should be used becomes the accepted ‘truth’ of what working in VR is.
We all need to think and dream about how we can best develop how we work in VR, otherwise we won’t all benefit.
Goal
The goal of this symposium is to spark dialogue around potential opportunities and issues of working with knowledge in VR.
Issues
Along with questions listed above and questions raised by you, there are two aspects which become prominent and underlay how we can develop VR environments which are open and connected:
• How addressability will work in VR; how locations, time, applications and locations in knowledge structures can be addressed. (you can’t address something if you can’t ‘address’ it). How will we move from one environment to another?
• Issues around infrastructures, ownership and compatibilities of knowledge products in different VR environments. Will we be able to take what we build in one environment into another, or will we have the same issues we had with compatibility we have experienced in traditional environments?
Record for the Future
To further the dialogue we will produce a useful record in the form of The Future of Text Vol III for practitioners, thinkers, students and the general public to reflect on for future efforts: Future Text Publishing
Join Us!
Join the conversation as we discuss different perspectives in how we can work in VR. To register please email frode@hegland.com describing your interest in text and VR.
Who We Are
This Symposium is hosted by Frode Alexander Hegland, Ismail Serageldin and Vint Cerf. The curators and editors of the event and the subsequent Future of Text book are Dene Grigar, Claus Atzenbeck and Mark Anderson. The Symposium is part of the work of the Future Text Lab.
Symposium Format
Paper. The format is for the presenter to submit a paper which will be distributed to all the attendees 1 week before the symposium, on the 20th of September, as a special issue of our Journal.
Presentation. On the day there will be just 5 mins for presentations, which may be exactly the same as the paper, a summary or a new presentation on the same topic, then 15 mins dialogue.
Information for presenters
When & Where
27th and 28th of September 2022, London, UK, and Online. iCal
Location to be announced.
Deeper literacy
13 July 2022
Thinking about using the deeper literacy angle to better promote what I am doing, including software:
A deeper literacy is needed for students to not be overwhelmed in today’s digital environment but to thrive. This will com
A deeper literacy also means developing tools which can make it possible.
To flourish in today’s complex digital world students will have to develop a deeper literacy.
We develop tools for a deeper literacy: Author and Reader for macOS
The deeper literacy students will need to thrive will require more powerful tools
More powerful tools for a deeper literacy.
Author helps students get to grips with their work in more powerful ways in a distraction free environment. This results in a deeper literacy where students can correctly cite with a copy & paste, instantly look up any information, visually map their concepts to better understand how they relate, view their work in different ways to generate different perspectives and export their final work with academic references automatically created.
Personal note
Yesterday Rodney was here. In summary: I asked him why he could not get in touch with Emily one or twice a week, he first started trying to list messages but we do have a record on Emily’s phone. He refused to answer, Emily got emotional, as did I. It ended up with the police when I assaulted him. I feel absolutely upset he was hurt, by me, but I would not have done anything different. He could not continue to torture Emily this way. Despite me assaulting him, the police were actually on my side. It’s horrible and I hope Emily can heal.
Tomorrow we are off for Estonia, for Alar & Kirstal’s wedding, which will be a big adventure!
Future of text ‘Relationship’s
We are having a themed discussion today, let’s see how it goes.
We decided to continue the discussion Monday. Here is the video: https://youtu.be/mN0fiNvWTSM
Thoughts
Thought, tweeted: Instead of just textual messages, how about contextual message? And what might that entail?
Another thought, tweeted: The most profound thing a human can do is to choose where to look from.
Deep Change for a Deeper Literacy
As I am sitting here on our flight to Estonia (and as I continue to write I the early mornings in the hotel room before the family wakes), I am wondering how to get Visual-Meta into wider use and how to get Author into wider use. I have had thoughts on this for a few weeks at a higher pitch than before and it goes in circles. However, as I was about to send an email to you Vint asking if we should try to make a plugin for web browsers to parse Visual-Meta and I think about making the BibTeX part of Visual-Meta larger so that users can copy it easily, manually and all sorts of ams thoughts, but then the question of what happens when reader software can parse Visual-Meta without authoring software and it becomes a bit weak. This is where it becomes clear that systematic change is necessary, so I have the following thoughts:
The name Web ‘browsing’ is apt since we do indeed surf the surface of the information we access. There is a current joke going around on social media that it used to be that people blamed lack of access to information when we make bad decisions and have poor communication, such as with climate change, political freedoms and actual war, then they kind of snicker and point to the Internet and the Web and say “well that wasn’t the reason”. A little funny, but information has always been available, what we need to keep updating is how we access the information and how we can orient ourselves within it. It has been said that one of the most profound things a person can do is to choose where to see from–what perspectives to access, not just what information to access.
This may sound overly ‘deep’ and though pretentious perhaps, and over-reaching, but deep is where we need to go. To deal with the amount of digital information we live in, we need to enable deeper literacies to develop. It is crucial to point out that digital information represents a new phase of rhetoric–whereas information (particularly recorded information) has always sought to persuade, it has done so passively. Much digital information is dynamic and actively seeks to influence the ‘reader’, not only social media but as part of layers of influence in the publishing and media industries, including academic.
As a society, or at least as the elements of society which cares about intellectual and political freedom and have an acceptance that our lives are very much shaped by the technologies we use to to think and communicate with, we have an obligation to take the possible futures of how we develop our connective tissues.
It will not be enough to invest in ‘high tech’ and ‘rich media’. We must also invest in the everyday tech of text. We will not get further towards our goal of enabling deeper literacies through only incremental feature updates to application software. We must upgrade the infrastructures of how information connects. One aspect of this is how documents can refer to each other and how reader software can understand the structures and semantics of documents. This is what Visual-Meta addresses. Another issue is how addressability itself can work, how something can refer to something else. It is easy to think this has been solved with Web Links but these are not permanent. Citations in academia help but they are not fully digital. Handles, such as DOI are a stop-gap but they are not permanent either and contact little useful metadata.
Photographers have ever improving camera systems and software through which they can edit their photographs. The Photoshop of the early 21st century could not do nearly as much as it can today, including ML powered subject extraction which nothing short of magic for the user, along with the ability to learn faces so that it becomes easy for a photographer to pull out only pictures of specific people and, because of the attached EXIF metadata, locations. These tools enable–and require–the user to learn how to use them however and photographers, along with videographers, game designers and gamers and 3d modellers do learn to use their tools in rich and expressive ways. This is not a new thing with digital technologies, just look at a sculptor at work, with a specific tool for ‘almost everything’. What is new is that this desire and ability to use tools is spreading deeper into the population, with social media giving people expressive tools as well, for how they present their experiences to others, with filters, layers and effects. We need to augment everybody but is impractical to try to upgrade those who do not feel the need, hence we need to present
Authors and reader are under serviced when it comes to how they can interact with the stuff of their work, words.
Not only are connections broken, how we can choose to view what is essentially an infinitely malleable medium, is hampered by our preconceptions of how text should look, based on centuries of analog, immobile text. To truly thrive in a world of fake news, social media companies which optimise interactions for their cashflow rather than for user’s information flow, masses of academic and scientific literature beyond anyone’s capacity to read it all, we need to augment the readers ability to get to grips with it all and the author’s ability to shape their thinning and communication clearly.
This is why we need to introduce Author and Reader to many more users as a proof of concept and hope that the Visual-Meta aspect will be copied by others. It is not enough to be able to produce Visual-Meta, the user will also need to have the means to use the data. I was thinking we could make a browser plugin for Visual-Meta to at least allows PDF’s and soon HTML to be augmented, but the interactions won’t be there and if the user copies text with Visual-Meta they won’t necessarily have anywhere to paste it. The whole system of reading and writing needs upgrading and that can only be done by upgrading the whole system.
Vint, what do you think? I cannot afford to invest any more family funds and I only make $500-$1,000 a month, which all goes into to development, which has been mostly paused over the last six months to a year costs $5,000 per month, for half-time work. Can you think of ways to fund this? There is so much polishing left to do and there are so many obvious improvements to make, including a richer Map view and Visual-Meta round trips to Reader to make reading more interactive and the assemble a basic Library feature. Costs for marketing comes on top of this, but only PR seems to work, not ads. We got a new review recently in Tech Republic and more would be useful but more platforms and more exposure would definitely be more useful.
I can make Author free, no problem, but there will always be updates for maintenance and improvement which I cannot afford to continue to build. Who might be interested in re-booting and upgrading reading and writing?
I would be very happy to support Google Docs use Visual-Meta (for pasting Visual-Meta copied citations and export), same with Microsoft Word and beyond.
This is the point in history where we can see back in time to almost the birth of the universe but we, as a species, cannot see the imminent danger of climate change, even as our cities boil. We must invest in how we think and communicate. Visual-Meta is part of this, just a small part, but hopefully a powerfully useful part if it is taken up by a sufficiently large number of users.
Mapping unknowns for connections (18 July 2022 chat)
For context and focus, what are the domains of discourse we want to study in this group?
Minds for motion through the world as we perceive it (Brandel comment, Frode ads therefore where we stand matters).
Tools change our affordances. Words in brain, paper, digital and VR are different.
Bob asks what size chunks of knowledge.
Manipulation of text in VR
Translation/Transfer to VR of text
Psychological considerations, issues re embodiment
What are the Tools?
Peter: https://web.archive.org/web/20050225144212/http://www.research.ibm.com/journal/sj/mit/tocpdf.html
Bob: How AI can produce good sub-heads?
Bob: What would we like to do which we cannot? Fx AI goes through a long document and assigns sub-heads.
Brandel: Write while walking.
Peter: Construct hypertext networks while walking. Store triples as audio at least.
Frode's referring to my convo with him about difference between writing (outward transmission) vs review/editing which means much closer interaction with the text and its structure. Silly thinks like it's much easier to proof large text and small text (IOW notions of ‘default' zoom might be flawed).
How to fold?
Frode: We have access, but do we have perspectives?
Author for Apple Watch: Siri, define ‘word’ as ‘definition’ and it appears in the Journal Defined Concepts, marked as voice created, ie • in front in the menu list.
Fabien: Video?! Players showing popular parts. Annotations for how things are read as well.
Mark: Makes me think of Chris Gutteridge’s of decomposable Glossaries – IOW the degree to which to drill in can reflect your facility with the subject under discussion.
Peter: Flashcards. To jog memory rather than store information. Linked in overview of the programmers apprentice. Dissolving distinctions between aliases and originals.
Fabien: I did it quite a bit https://fabien.benetou.fr/Wiki/MemoryRecipe within my own wiki but I didn't put the time so I assume it means I didn't find that so useful
Mark: Regardless of [brandname] it seems the challenge is taking to salient points of a doc (assumed gathered for us) and passing it in a form a flash card app can consume.
Peter: aTbRef flashcard set!
Bob: Argument mapping by Akamatsu.
Frode: Argument mapping as logos. Also need mythos. How can we work on changing how we stand?
Mark: Re just throwing stuff onto the map (i.e. environment/wall/etc.) here's an old Tinderbox exploration. BTW, what exactly I was exploring, I forget, but the point is to illustrate to early phase of exploration, where I'm deliberately avoiding premature formalisation.
Jim presents a 6 sided cube with perspectives. We then discuss how this can be used, partially in VR.
Frode: How to read with large volumes of text/documents? Imagine defining specific texts as democratic, authoritarian and so on, then using ML to present new texts in different contexts.
Fabien: https://twitter.com/utopiah/status/1541446650316595200 sharing perspective in 360s and an earlier exploration which was supposed to add recording audio from someone else to replay https://vatelier.benetou.fr/MyDemo/PerspectiveNetwork/Perspectives/
Mark: In the context of perspectives, where do the unknowns go, indeed the unknown unknown. Like Dark Matter the latter are also part of the picture.
IIRC this (unknown unknowns) relates to the Johari Window: https://en.wikipedia.org/wiki/Johari_window
Email to Bruce Horn regarding use of ML with text:
Bruce, In follow up to our text chat: I just used my Apple Watch to dictate this message to our mutual friend Kevin today: “I am by the ports in Tallinn in Estonia, taking pictures of all ships, wish you were here” This is of course quite extraordinary precision, almost perfect. ML has become real and useful, and also vastly overblown, depending on who we listen to. In Author we use Apple libraries to have a view to show only Names (plus headings for context) which is basic ML of course. Apple has a few ML libraries available: https://developer.apple.com/machine-learning/ A few dream scenarios the group has been talking about:
What do you think, are any of these, or similar, feasible?
AI & VR for The Future of Text at Southampton (Vint Email)
It struck me that AI, at least in the form of ML, has become very useful and usable. When something rolls out of the research lab and into everyday use it’s easy to forget how fantastic it is, it’s potential and–specifically for us developers–that it is now something which can be developed without a massive overhead. This is exciting and meaningful. Having had covid over the first part of our trip to Estonia and spent quite a few hours on YouTube watching random things to distract me from the nausea, a few of the videos I came across was about ML for writing†, including sites to: Generate Speech From Text; Rytr: Writer Better and Faster, Talk To Books (Google AI Experiment), Thing Translator (Google AI Experiment); Replika: AI-Powered Companion; WordTune: Say what you mean easier, many of which are about text.
Considering that we use Apple’s speech to text interface in Author and we have views based on named entities, which uses basic ML, we, as a small developer should start looking into AI for text more. My goal has never simply been to make a commercial project and sell it against the competition however, so any AI benefits should ideally be packed in a way for other text systems to use.
A key point has been that there are so many ‘cool’ AI text services available, but they are scattered.
I first thought of a goal of selling Author to Apple as a ‘pro’ application, alongside Final Cut and Logic, leaving Pages as a general typing and layout app. This is of course a lofty ambition and for Apple to be at all interested, Author would have to be quite impressive. It is not impressive to this level yet of course. With an architecture to allow for rich customisation and plugins of AI and VR (to access aspects of documents easily in VR), Author might start to become something good enough though. I can imagine–OK, dream–of Tim Cook introducing:
“an addition to our Pro apps: ‘Author’. Not only is Author optimised for the latest M series chips and runs on Mac, iOS, iPadOS, RealityOS, and even to an extent WatchOS, featuring a minimalist workspace optimised for professional authors of books, websites, blogs and academic documents, Author features advanced AI tools. Let me show you a few [demo]. In addition to what you can do in Author though, all the power behind Author is available for developers to take advantage of. As well as being a Pro app, Author is also a technology demonstrator. After all, the future of text is important enough to build together”
So this is my dream (in very truncated form). I think it could be a very useful step on our work on the future of text. If Google would prefer to buy it and do the same kind of model that would of course be interesting as well. However, I am a bit of an Apple fanboy and one of my closest friends, Bruce Horn (who wrote the original Macintosh Finder) is back at Apple, working on Siri/AI and Brandel whom I see twice a week is there working on VR.
AR multiplied by VR can really propel the future of text into a powerful augmentation I think.
A step before this would likely be to get more of a track record and network. This is why I have started reading more about the realities of present and near-present AI and this is why I think we need more AI people at the Future of Text Symposium.
It is also why I think we should try to launch a small research lab at Southampton, for VR&AI for Text. Dave Millard has tentatively agreed it makes a lot of sense and Southampton is very interested in AI, with Wendy being a huge AI persona. I don’t know anything more about how this should be done, at this stage, but it could be based around the notion of becoming a hub for text AI research as well as a resource for getting AI components for other projects. Depending on level of funding, we could integrate what we find and learn into Author, but also put the code and method on a repository for anyone to use. There are two very separate but connected strands here I think: The AI tech and the way it’s integrated into the writing app–the interface to the commands and how they relate to the rest of the user’s workflow. Almost anyone can make the best tool for a single command, but it gets challenging when this needs to be integrated into a system, a workflow, both in terms of how it might fit on a screen–the interface–and how it will fit in the user’s mental models and workflow–‘behind the face’. This is where I want to compete. The AI and the research findings should be open.
If I present this to Wendy, and yes, I’d be happy to work on this under her, would you support it by being part of it? I will ask Ismail also of course. Do you think there would be some Google research money available for this?
At this point, what matters for me is to not be stuck in the past but really try to use the opportunities AI and VR offers for future writing and reading, not only manual direct controls on 2D surfaces.
What do you think, try for a Southampton Lab?
Or try to get funding to improve and market Author more?
Or go for the bigger idea of a more funded research centre?
I am eager to learn what you think. I am out of covid now, just a bit dizzy and some back pain, and in London Tuesday.
AI to help writing
How about cmd-/ (?) in Author to automatically show text in three panels to suggest further writing, based on what the user has already typed but also, in panel 1 based on the user’s previous work, panel 2 general internet info and 3 based on what the user has cited plus a specific collection of documents from their field?
Update superscript for Endnotes
Instead of superscript a,b,c,d,e… and then aa, bb, and so on, we need to be more clever with the sequencing otherwise they get long fast. Can we do a, then b and so on, and when we go into two letters aa followed by ab then ac and ad and so on? And then continue with aaa after all of those have been exhausted, starting then with aaaa and aaab then aaac and so on?
A digital game keeps track of the rules (29 July 2022)
Brandel comment on games and how in the real world we have to keep our own rules. Roughly 1:38† mins into the discussion.
In terms of rules, I guess the ‘ideal' would be a fantastic computer game type environment for information views and connections which are clear, but we can always, Matrix style, exit the system to change the rules. Make sense?
Mac Meeting Recording app
Here I am in my Journal thinking about how I’d like to use transcripts from recorded meetings.
I’d like to have a local app which records audio locally and transcribes.
Recording
It would be very useful if it can differentiate between audio which comes from the computer and audio which does not, so that it will know when I am speaking and when others are speaking. It would be very good if it can record on multiple computers and iCloud synch so that there is no question who said what and all speech is labeled correctly and without user effort. At the end of a recording, the user can click Stop and also Share, which will work by something as simple as adding a URL once, provided by the host, and then they can choose to share to that URL. There will need to be some management for this of course, with multiple places to share to and to edit these.
Live use
It will provide transcripts live in a window which does not need to be seen all the time so that it can be used for short term recall but not as a distraction.
Finding Specific Phrases
There will be a search function for phrases which can be filtered by speakers, time of day and date range, in a very simple interface to set these parameters (keyword search box visible at all times, a list of speakers in the system as buttons to mute any at the bottom and a reveal for the rest for more advanced use).
Browsing
The presentation of the chat is a regular Author document so that the user can fold and find and highlight and so on.
Citing
This is where it gets tricky and I have to think some more.
Summary
Having gone though this basic exercise I think the human part needs to be augmented rather than automated, to use that phrase. When Brandel started talking about games I should be able to write a note on this, as I did in the section above, but then through some mechanism have it linked to the video when later uploaded. The connective mechanism is time. If Author knows what time I wrote something, because I chose to insert a timestamp, then I simply need a way for Author to know later, when I have the URL to the YouTube video, what time the YouTube video started recording. It does not have to be to the millisecond but meetings tend to start on the hour or half past. Imagine this simple scenario: I write a note, then timestamp it (cmd-t where there is now an option to ‘Insert Timestamp’ (along with ‘Search Books’ and ‘Use YouTube URL’). This simply adds a character, much like the character for endnote, which turns into a date and time stamp on export to PDF, but more importantly, it does this: When user clicks on it the user gets a dialog with the options to:
If the user chooses to paste the YouTube URL of the recording of that meeting, the user can also choose to specify the time of day it started, to know what the offset would be. This will be remembered by the system, for the next time, based on the current document (since there may be different documents for different meetings).
Once the user clicks ‘OK’, the system converts the YouTube URL into a link inside the video and goes online and retrieves the transcribed text… ok, stop, this is getting messy again.
Simple Author Recording System
Let’s try a new version: I am in Author, I choose to ‘Record’ and while I do so, I can choose to insert a key moment so I might type a keyword, such as ‘Brandel said…’ or simply do cmd-[ and the the recording is marked for that point and the characters [] are entered at the cursor position.
Any time later I can click on it and it opens into a dialog where the transcribed text is shown, including 15 seconds before I clicked the button and 1 min after. I then choose the in and out points and ‘OK’. And here is the neat thing, this acts as stretch text when in regular view: I can trim what transcribed text is to be shown, by moving the [ and the ] on the page. When double clicked, I get the full edit window where I can edit the text, including selecting sections and then clicking on a person’s name (shown in a button at the bottom of the dialog, built up over time, easy to add by clicking a + button) and so on.
The recording is also uploaded to YouTube automatically after each section (with a card image, since YouTube does not accept audio only), with the system knowing the URL to the recording.
On export the [] is turned into “quotes” with a URL to the YouTube audio section right after.
There will also be a dialog for searching all recordings, this might be in the Journal or a new space called ‘Recordings’ where the user can interact with the recording transcript like a normal Author document to get all the search and view capabilities. Doug thought it important to always have all the affordances in every view. How audio and text will be shown is an issue to be worked on. The primary use for this is initially to capture useful moments at the time. Secondary use will be to go through the recording later for keywords etc. I can imagine though, in the ‘Recordings’ view, that there is a play bar at the top of the screen, over the transcribed text, where the user can set if it’s only one session or all of them or a data range.
Notes
If I forget to record, and there is 5 min with no speaking, the system stops recording.
The recordings are kept outside the document, in a clearly marked folder in the Author documents folder.