[GH-ISSUE #127] Writing to disk in real-time

kerem commented

2026-02-25 20:32:35 +03:00

Owner

Originally created by @vvv on GitHub (Aug 19, 2015).
Original GitHub issue: https://github.com/asciinema/asciinema/issues/127

Steps to reproduce:

start recording to a file;
kill asciinema process (e.g., close the terminal window);
try to locate the recording — there is none.

See the demo.

It would be very useful if asciinema flushed the recording file periodically. Unfinished JSON file is easy to fix (asciinema play could do this). The loss of a session recording may be rather disappointing at times.

(My dayjob project has long-running system tests. I use asciinema to record their execution, because its web UI allows me to jump to interesting spots and fast-forward the test.)

Originally created by @vvv on GitHub (Aug 19, 2015). Original GitHub issue: https://github.com/asciinema/asciinema/issues/127 Steps to reproduce: - start recording to a file; - kill asciinema process (e.g., close the terminal window); - try to locate the recording — there is none. See [the demo](https://asciinema.org/a/4tzbj4fmp5f1pl18n0l07d311). It would be very useful if asciinema flushed the recording file periodically. Unfinished JSON file is easy to fix (`asciinema play` could do this). The loss of a session recording may be rather disappointing at times. (My dayjob project has long-running system tests. I use asciinema to record their execution, because its web UI allows me to jump to interesting spots and fast-forward the test.)

kerem

2026-02-25 20:32:35 +03:00

closed this issue
added the
feature request

improvement
labels

kerem commented

2026-02-25 20:32:35 +03:00

Author

Owner

@ku1ik commented on GitHub (Aug 28, 2015):

Good catch @vvv. Looks like it can be fixed rather easily.

@ku1ik commented on GitHub (Aug 28, 2015): Good catch @vvv. Looks like it can be fixed rather easily.

kerem commented

2026-02-25 20:32:35 +03:00

Author

Owner

@ku1ik commented on GitHub (Jan 31, 2016):

I looked at this. Current thoughts:

asciinema doesn't generate the JSON stream on the fly - it generates whole JSON string and saves it to file once it has the whole stdout captured. This is mostly because how Go's JSON marshaller works.

One possibility is to flush to a tmp file (foo.json.tmp if you record to foo.json). It could look like:

{ "version": 2, "width": 80, "height": 24, "env": { "TERM": "xterm-256color" } ... }
[0.1, "bash-4.3$ "]
[0.3, "l"]
[0.1, "s"]
...
...

Basically a JSON file with multiple top-level objects, first being JSON map with metadata, all the rest being stdout prints that should eventually end up under stdout key in the final file.

As for recovering, you could then easily do it yourself (just moving lines around in vim).
There could also be a new command asciinema recover foo.json.tmp foo.json to automate this (not sure if making the recovery a part of asciinema play is the best).

@ku1ik commented on GitHub (Jan 31, 2016): I looked at this. Current thoughts: asciinema doesn't generate the JSON stream on the fly - it generates whole JSON string and saves it to file once it has the whole stdout captured. This is mostly because how Go's JSON marshaller works. One possibility is to flush to a tmp file (`foo.json.tmp` if you record to `foo.json`). It could look like: ``` json { "version": 2, "width": 80, "height": 24, "env": { "TERM": "xterm-256color" } ... } [0.1, "bash-4.3$ "] [0.3, "l"] [0.1, "s"] ... ... ``` Basically a JSON file with multiple top-level objects, first being JSON map with metadata, all the rest being stdout prints that should eventually end up under `stdout` key in the final file. As for recovering, you could then easily do it yourself (just moving lines around in vim). There could also be a new command `asciinema recover foo.json.tmp foo.json` to automate this (not sure if making the recovery a part of `asciinema play` is the best).

kerem commented

2026-02-25 20:32:35 +03:00

Author

Owner

@vvv commented on GitHub (Feb 4, 2016):

One possibility is to flush to a tmp file (foo.json.tmp if you record to foo.json).

Sounds good! This feature would resemble Emacs' auto-save files (#foo.json#) and Vim recovery files (.foo.json.swp.)

As for recovering, you could then easily do it yourself (just moving lines around in vim).

Frankly, I'd prefer "recover" command, or even a lightweight -r | --recover option, to manual fiddling with an auto-save file.

@vvv commented on GitHub (Feb 4, 2016): > One possibility is to flush to a tmp file (`foo.json.tmp` if you record to `foo.json`). Sounds good! This feature would resemble Emacs' [auto-save files](https://www.gnu.org/software/emacs/manual/html_node/emacs/Auto-Save-Files.html) (`#foo.json#`) and Vim [recovery files](http://vimdoc.sourceforge.net/htmldoc/usr_11.html) (`.foo.json.swp`.) > As for recovering, you could then easily do it yourself (just moving lines around in vim). Frankly, I'd prefer "recover" command, or even a lightweight `-r | --recover` option, to manual fiddling with an auto-save file.

kerem commented

2026-02-25 20:32:35 +03:00

Author

Owner

@xloem commented on GitHub (Nov 6, 2016):

Just to state this overtly, asciinema/discussions#10 would fix this issue as well.

@xloem commented on GitHub (Nov 6, 2016): Just to state this overtly, asciinema/discussions#10 would fix this issue as well.

kerem commented

2026-02-25 20:32:35 +03:00

Author

Owner

@ku1ik commented on GitHub (Nov 8, 2016):

@xloem asciinema/discussions#10 applies only to the case when you rec+upload to asciinema.org in one step. asciinema rec demo.json also allows you to record to a file without automatically uploading to the site, and incrementally writing to disk in this case would be equally useful.

@ku1ik commented on GitHub (Nov 8, 2016): @xloem asciinema/discussions#10 applies only to the case when you rec+upload to asciinema.org in one step. `asciinema rec demo.json` also allows you to record to a file without automatically uploading to the site, and incrementally writing to disk in this case would be equally useful.

kerem commented

2026-02-25 20:32:35 +03:00

Author

Owner

@ku1ik commented on GitHub (Mar 14, 2017):

I started working on a draft of asciicast v2 format in asciinema/asciinema#196, which should nicely solve writing in real-time to disk (and pipe, network, what have you).

Direct link to doc from this PR: https://github.com/asciinema/asciinema/blob/asciicast-v2/doc/asciicast-v2.md

Feedback highly appreciated.

@ku1ik commented on GitHub (Mar 14, 2017): I started working on a draft of `asciicast v2` format in asciinema/asciinema#196, which should nicely solve writing in real-time to disk (and pipe, network, what have you). Direct link to doc from this PR: https://github.com/asciinema/asciinema/blob/asciicast-v2/doc/asciicast-v2.md Feedback highly appreciated.

kerem commented

2026-02-25 20:32:35 +03:00

Author

Owner

@vvv commented on GitHub (Mar 15, 2017):

@sickill I don't see how the change of file format (NDJSON instead of JSON) applies to incremental update of output file. Could you explain?

@vvv commented on GitHub (Mar 15, 2017): @sickill I don't see how the change of file format ([NDJSON](https://github.com/ndjson/ndjson-spec) instead of JSON) applies to incremental update of output file. Could you explain?

kerem commented

2026-02-25 20:32:35 +03:00

Author

Owner

@ku1ik commented on GitHub (Mar 15, 2017):

@vvv in normal JSON file you have a single object and you can't write to it incrementally, it has to be written as a whole (technically you can but if you crash etc then you end up with invalid JSON file, missing the closing brackets). With NDJSON (or JSONLines which is almost identical) you have multiple JSON objects in a single file, each on its own line. So you can be appending new lines with new data and stop/crash at will and never leaving the file invalid.

@ku1ik commented on GitHub (Mar 15, 2017): @vvv in normal JSON file you have a single object and you can't write to it incrementally, it has to be written as a whole (technically you can but if you crash etc then you end up with invalid JSON file, missing the closing brackets). With NDJSON (or [JSONLines](http://jsonlines.org/) which is almost identical) you have multiple JSON objects in a single file, each on its own line. So you can be appending new lines with new data and stop/crash at will and never leaving the file invalid.

kerem commented

2026-02-25 20:32:35 +03:00

Author

Owner

@ku1ik commented on GitHub (Mar 15, 2017):

I've updated the draft of asciicast v2 format doc to make it more clear regarding motivation / what problems it solves.

@ku1ik commented on GitHub (Mar 15, 2017): I've updated the draft of asciicast v2 format doc to make it more clear regarding motivation / what problems it solves.

kerem commented

2026-02-25 20:32:35 +03:00

Author

Owner

@josegonzalez commented on GitHub (Apr 25, 2017):

@sickill one thing that would be good would be to have the start time of the session stored in the initial metadata. This could be extracted in other tooling to provide auditable ssh sessions.

Additionally, being able to "inject" metadata might be interesting, so you could potentially tag a created session with stuff like:

user that ssh'd
hostname
server environment

And have that exposed in some external ui.

EDIT: Seems there is a PR for the format, so I'll comment there :)

@josegonzalez commented on GitHub (Apr 25, 2017): @sickill one thing that would be good would be to have the start time of the session stored in the initial metadata. This could be extracted in other tooling to provide auditable ssh sessions. Additionally, being able to "inject" metadata might be interesting, so you could potentially tag a created session with stuff like: - user that ssh'd - hostname - server environment And have that exposed in some external ui. > EDIT: Seems there is a [PR](https://github.com/asciinema/asciinema/pull/196) for the format, so I'll comment there :)

kerem commented

2026-02-25 20:32:35 +03:00

Author

Owner

@guyromm commented on GitHub (Apr 25, 2017):

i currently have asciinema set up for session logging of remote employees
connecting via ssh to a secure net via a jumphost. being able to save,
stream and replay sessions, as they occur, would greatly enhance the
usefulness of asciinema in said scenario.

On Tue, Apr 25, 2017 at 5:17 AM, Jose Diaz-Gonzalez <
notifications@github.com> wrote:

@sickill https://github.com/sickill one thing that would be good would
be to have the start time of the session stored in the initial metadata.
This could be extracted in other tooling to provide auditable ssh sessions.

Additionally, being able to "inject" metadata might be interesting, so you
could potentially tag a created session with stuff like:

user that ssh'd

hostname

server environment

And have that exposed in some external ui.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/asciinema/asciinema/issues/127#issuecomment-296872546,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAi2o-cZFmoJOnPeabG0UkPfb8MVR3EMks5rzVfNgaJpZM4FuWm_
.

@guyromm commented on GitHub (Apr 25, 2017): i currently have asciinema set up for session logging of remote employees connecting via ssh to a secure net via a jumphost. being able to save, stream and replay sessions, as they occur, would greatly enhance the usefulness of asciinema in said scenario. On Tue, Apr 25, 2017 at 5:17 AM, Jose Diaz-Gonzalez < notifications@github.com> wrote: > @sickill <https://github.com/sickill> one thing that would be good would > be to have the start time of the session stored in the initial metadata. > This could be extracted in other tooling to provide auditable ssh sessions. > > Additionally, being able to "inject" metadata might be interesting, so you > could potentially tag a created session with stuff like: > > - user that ssh'd > - hostname > - server environment > > And have that exposed in some external ui. > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <https://github.com/asciinema/asciinema/issues/127#issuecomment-296872546>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AAi2o-cZFmoJOnPeabG0UkPfb8MVR3EMks5rzVfNgaJpZM4FuWm_> > . >

kerem commented

2026-02-25 20:32:35 +03:00

Author

Owner

@gizmonicus commented on GitHub (Jun 15, 2017):

I use asciinema (well, not anymore, I'm reverting back to script because of this issue) to keep a record of all my terminal sessions to CYA as well as for reference if I forgot what I did on server X last week at 3 pm. Problem is, since this only flushes on successful exits, half my sessions have no data. Closing a window for a hung session for example, causes a total loss of the session recording. This was unexpected behavior and quite disappointing.

@gizmonicus commented on GitHub (Jun 15, 2017): I use asciinema (well, not anymore, I'm reverting back to script because of this issue) to keep a record of all my terminal sessions to CYA as well as for reference if I forgot what I did on server X last week at 3 pm. Problem is, since this only flushes on successful exits, half my sessions have no data. Closing a window for a hung session for example, causes a total loss of the session recording. This was unexpected behavior and quite disappointing.

kerem commented

2026-02-25 20:32:36 +03:00

Author

Owner

@ku1ik commented on GitHub (Jun 16, 2017):

@gizmonicus asciicast v2 format and writing to disk in real-time is the highest priority for the next release. No ETA, but that's gonna happen soon.

@ku1ik commented on GitHub (Jun 16, 2017): @gizmonicus asciicast v2 format and writing to disk in real-time is the highest priority for the next release. No ETA, but that's gonna happen soon.

kerem commented

2026-02-25 20:32:36 +03:00

Author

Owner

@gizmonicus commented on GitHub (Jun 16, 2017):

@sickill - Thank you, this is good to hear. I'll keep an eye on it. I really would prefer asciinema to script because I can embed the content in our Wiki for other sysadmins.

@gizmonicus commented on GitHub (Jun 16, 2017): @sickill - Thank you, this is good to hear. I'll keep an eye on it. I really would prefer asciinema to script because I can embed the content in our Wiki for other sysadmins.

kerem commented

2026-02-25 20:32:36 +03:00

Author

Owner

@IonicaBizau commented on GitHub (Sep 1, 2017):

@timofonic You can ask your question here and wait patiently. There is no need to spam everybody by pinging users this way.

@IonicaBizau commented on GitHub (Sep 1, 2017): @timofonic You can ask your question here and wait patiently. There is no need to spam everybody by pinging users this way.

kerem commented

2026-02-25 20:32:36 +03:00

Author

Owner

@ku1ik commented on GitHub (Sep 3, 2017):

Update: writing to disk in real-time has been implemented in asciinema/asciinema#196 and will be part of the next release.

If you'd like to beta-test it then checkout develop branch and run it from the checkout dir with: python3 -m asciinema rec filename. I'd really appreciate feedback on how it works on various Linux distros and macOS 👋

Note: as of today the server and the web player don't support new asciicast format so it can be used for local recording and in-terminal playback only.

@ku1ik commented on GitHub (Sep 3, 2017): Update: writing to disk in real-time has been implemented in asciinema/asciinema#196 and will be part of the next release. If you'd like to beta-test it then checkout `develop` branch and run it from the checkout dir with: `python3 -m asciinema rec filename`. I'd really appreciate feedback on how it works on various Linux distros and macOS 👋 Note: as of today the server and the web player don't support new asciicast format so it can be used for local recording and in-terminal playback only.

kerem commented

2026-02-25 20:32:36 +03:00

Author

Owner

@metasoarous commented on GitHub (Sep 12, 2017):

Great work @sickill! I just tested this out and was able to get it to work (Ubuntu 17.04).

One bit of clarification though; What triggers the dumps? Is it file buffering or a timeout? Either way, what are the specific thresholds? I ask because after doing a couple of quick echos, I didn't see anything in the file, but as I played around with it more, stuff started showing up.

@metasoarous commented on GitHub (Sep 12, 2017): Great work @sickill! I just tested this out and was able to get it to work (Ubuntu 17.04). One bit of clarification though; What triggers the dumps? Is it file buffering or a timeout? Either way, what are the specific thresholds? I ask because after doing a couple of quick echos, I didn't see anything in the file, but as I played around with it more, stuff started showing up.

kerem commented

2026-02-25 20:32:36 +03:00

Author

Owner

@ku1ik commented on GitHub (Sep 12, 2017):

@metasoarous that's a great question.

There's no explicit timeout or any other mechanism to batch the writes, it goes in real-time through queue:

github.com/asciinema/asciinema@8e1b6f7da1/asciinema/asciicast/v2.py (L72)

to separate "writer" process:

github.com/asciinema/asciinema@8e1b6f7da1/asciinema/asciicast/v2.py (L36-L40)

which does f.write(...).

Having said that, I also observed that it's written in ~8KB chunks, so there's definitely some buffering going on here. I suspect it's Python's TextIOWrapper being responsible here.

What do you think would best here? We can turn off buffering on the opened io object, or implement explicit trigger, either f.flush() on every write, or byte/time counting and flushing when threshold is hit.

@ku1ik commented on GitHub (Sep 12, 2017): @metasoarous that's a great question. There's no explicit timeout or any other mechanism to batch the writes, it goes in real-time through queue: https://github.com/asciinema/asciinema/blob/8e1b6f7da1a0ad4d52e63998b14c1a5133fc7836/asciinema/asciicast/v2.py#L72 to separate "writer" process: https://github.com/asciinema/asciinema/blob/8e1b6f7da1a0ad4d52e63998b14c1a5133fc7836/asciinema/asciicast/v2.py#L36-L40 which does `f.write(...)`. Having said that, I also observed that it's written in ~8KB chunks, so there's definitely some buffering going on here. I suspect it's Python's [`TextIOWrapper`](https://docs.python.org/3/library/io.html#io.TextIOWrapper) being responsible here. What do you think would best here? We can turn off buffering on the opened io object, or implement explicit trigger, either `f.flush()` on every write, or byte/time counting and flushing when threshold is hit.

kerem commented

2026-02-25 20:32:36 +03:00

Author

Owner

@metasoarous commented on GitHub (Sep 13, 2017):

I'd imagine turning off buffering on the io object might be the easiest solution, with manual flushes every N seconds a close second. I could imagine fancier strategies, but something close to one of those choices ought to do the trick.

@metasoarous commented on GitHub (Sep 13, 2017): I'd imagine turning off buffering on the io object might be the easiest solution, with manual flushes every N seconds a close second. I could imagine fancier strategies, but something close to one of those choices ought to do the trick.

kerem commented

2026-02-25 20:32:36 +03:00

Author

Owner

@ku1ik commented on GitHub (Sep 13, 2017):

I changed it so it now explicitly sets buffering policy to "line buffering" (== flush when write includes \n, which is the case for 100% of writes in our case) when opening the file for writing. If you pull and try it now you should be seeing file growing immediately.

@ku1ik commented on GitHub (Sep 13, 2017): I changed it so it now explicitly sets buffering policy to "line buffering" (== flush when write includes `\n`, which is the case for 100% of writes in our case) when opening the file for writing. If you pull and try it now you should be seeing file growing immediately.

kerem commented

2026-02-25 20:32:36 +03:00

Author

Owner

@metasoarous commented on GitHub (Sep 13, 2017):

Wonderful! Thanks so much for your rapid attention!

@metasoarous commented on GitHub (Sep 13, 2017): Wonderful! Thanks so much for your rapid attention!

kerem commented

2026-02-25 20:32:36 +03:00

Author

Owner

@ku1ik commented on GitHub (Sep 16, 2017):

Given this has been implemented (in asciinema/asciinema#227) and will be released with the upcoming asciinema 2.0 I'm closing this issue.

@ku1ik commented on GitHub (Sep 16, 2017): Given this has been implemented (in asciinema/asciinema#227) and will be released with the upcoming asciinema 2.0 I'm closing this issue.

kerem commented

2026-02-25 20:32:36 +03:00

Author

Owner

@ku1ik commented on GitHub (Sep 16, 2017):

Note: if anyone wants to try this then checkout v2 branch.

@ku1ik commented on GitHub (Sep 16, 2017): Note: if anyone wants to try this then checkout `v2` branch.

kerem referenced this issue

2026-03-15 08:52:12 +03:00

[GH-ISSUE #96] Ability to convert recorded sessions to different formats #690

Rows
Columns

[GH-ISSUE #127] Writing to disk in real-time #96