ParallelTalk MVP v1 Implementation

from ParallelTalk MVP v1

20230602

• I think I can finish it in a day if I put my mind to it, so I’ll build it while I’m in the mood (blu3mo)
• https://www.100ms.live/docs/javascript/v2/how-to-guides/record-and-live-stream/hls/hls
• https://github.com/video-dev/hls.js/
• 
  • For now, I can display webrtc and hls at the same time, but the delay and instability of hls are severe.
  • I need to think of a way to reduce the delay.
    • Beam -> ?? -> HLS, that flow.
  • I need to think of another way to obtain past footage.
    • Record locally?
      • This might work.
    • Can’t I take the buffer on the central server with WebRTC?
    • YouTube Stream API
      • This is probably just for the prototype.
      • I’ll explore the state of the YouTube Stream API while creating Work Delivery to Future Self.
  • 
    • It’s really expensive.
    • I guess I can’t rely on 100ms for this.
  • https://chat.openai.com/share/7ab91389-2740-4d11-a5a3-dc1b5645e0a0
    • It seems like having it locally might work (blu3mo)(blu3mo)
    • MediaRecording API
    • Flickering occurs.
      • https://www.tilcode.com/react-fixing-flickering-video-redraw/

      • Use a double buffering technique: Create two video elements and switch between them while updating the source of the non-visible one. This way, the update won’t be visible to the user, and you can avoid the flickering effect.

        • This is bad because PiP will break.

      • Update the video src less frequently: Instead of updating the src every 5000ms, try increasing the interval to a larger value. This will reduce the number of times the video flickers, although it may not completely eliminate the issue.

      • Concatenate the chunks before updating the src: Instead of updating the src with the latest chunk, try concatenating all the chunks and updating the src once the recording is stopped. This way, the flickering will only happen once, at the end of the recording.

      • Use a custom video player: Utilize a custom video player like Video.js or Plyr with better buffer handling capabilities. These players may provide a smoother playback experience and handle the src updates more efficiently, reducing the flickering effect.

        • It’s better to have more freedom in the future, so maybe I’ll try Video.js (blu3mo)

      • Implement a custom buffer management system: Instead of relying on the browser’s default buffer handling, implement a custom buffer management system that can handle the src updates more efficiently. This may involve using a combination of the MediaSource API and SourceBuffer API to manage the video playback and reduce flickering.

        • This is it (blu3mo)
        • https://stackoverflow.com/questions/50333767/html5-video-streaming-video-with-blob-urls/50354182
        • https://qiita.com/soebosi/items/74464d2f34d7416336eb

20230304

• I have a clear image of the MVP I want to create, so I want to implement it over the weekend.

  • Since it’s a weekend with no assignments, I want to make good use of it.
  • Hackathon vibes.

• Creating Things That Work Even When Small

• If I can make it this far, I can say I’ve achieved my goal.

• Sat

  • I want to aim for at least 1, preferably 2.
  • 1 done, it was just running the demo of 100ms ✅

• Sun

  • If 2 is properly done, 3, 4, 5 will just be a matter of doing them.
    • I’m a little worried about the specifications of 100ms for 3.

• Something that can make calls and listen to past audio.

  • HLS streaming
    • https://www.100ms.live/blog/hls-streaming
      • It’s very comprehensive.

    • 100ms supports live streaming output via HLS. However, we use WebRTC as input for the stream, unlike services that offer the infrastructure for live streaming alone, which generally use RTMP.

      • I see~ (blu3mo)
      • Originally, I was planning to do this with Kineto, but I changed it to RMTP->HLS along the way.
    • 100ms will handle video recording for up to 72 hours.
      • Beyond that, you need to integrate with AWS.
  • SFU Recording
    • When doing WebRTC via the central server instead of p2p, it’s the one that records it.
    • Either way, I need to stream it with HLS, so I can leave this for now.

• In fact, 100ms-web has too many features and is difficult to handle.

  • I want to work with something simpler.
  • https://github.com/100mslive/clubhouse-clone-react- This seems good among the simple ones, right?

• https://www.100ms.live/blog/clubhouse-clone-react

• Will this work?

20230131

• https://www.100ms.live/examples/live-streaming-starter-kit
• This seems to be a good base (blu3mo).
• “Creating a branch”
• Whether you are alone or there are two people at HEAD, internally it is the same state.
• It seems good to cut out the blank parts during playback.
• There was a book at Technical Book Festival that introduced an algorithm to cut out the blank parts (u7693).
  • Voicy Tech Story vol.4
  • https://techbookfest.org/product/dAJMsLnuGbnRYwQHX1sW9z
• As an implementation, it would be easy and bandwidth-friendly to have everyone in the same WebRTC room.
• When creating a branch, adjust the volume on the client side (?) so that you can’t hear each other.
• It seems possible to achieve 100ms, making it impossible to hear a specific person’s voice (blu3mo).
• It’s beyond MVP, but ParallelTalk, an effect where you can faintly hear the voices of other branches also seems good.