3D Gaussian splatting environment enhanced with AI powered chatbot NPCs

Hello, my name is Benoît, and I’m looking for a talented Unity game developer to join my project team. The endeavor involves creating an interactive non-fictional experience which will be distributed as an installation. Utilizing Unity3D and specialized plugins like Gaussian Splatting and InWord.ai, the project aims to construct a visually stunning environment where AI-powered NPCs (trained on real people dialogues) interact with players. For more details, please see the information below.

Responsibilities:

1. Environment Development:
   • Utilize Unity3D and specialized plugins like Gaussian Splatting to create visually stunning environments.
   • Composite 3D Gaussian Splatting environments and individual scans of people into the Unity project. (Integrate Vologram dynamic volumetric capture as an alternative to static Gaussian Splattings).

2. NPC Integration:
   • Integrate intelligent ChatBot cloned characters from InWorld.ai using real-world conversations to train ChatBots.
   • Match ChatBot conversational NPCs to each scan of individual people for an engaging audio experience.

3. UI/UX Design:
   • Design and implement a user-friendly UI/UX for a single-player experience using a smartphone as a joystick.
   • Ensure seamless microphone functionality for interacting with cloned NPCs.
   • Debug and optimize UI/UX elements for smooth navigation and gameplay.

Qualifications:

1. Technical Proficiency and Capacity to research new developments:
   Strong proficiency in Unity3D development, specially experience with non-traditional graphics enabled through game engine plugins coming from GitHub, and AI integration, particularly with ChatBot systems and real-world data training such as OpenAI and InWorld.ai.

2. UI/UX Expertise:
   Demonstrated expertise in UI/UX design principles and implementation within Unity projects, with the ability to create intuitive and visually appealing user interfaces for interactive experiences.

3. Problem-Solving Skills:
   Excellent debugging and troubleshooting skills to address technical challenges and optimize performance, coupled with a proven ability to innovate and explore new features or enhancements to improve the overall user experience.

4. Collaborative Team Player:
   Strong communication skills and the ability to collaborate effectively with multidisciplinary teams.

5. Passion for Gaming and Innovation:
   Enthusiasm for pushing the boundaries of gaming technology and creating photo realistic, non-fictional experiences, along with a demonstrated interest in exploring new tools, technologies, and artistic inspirations to achieve realism and engagement.

Additional Information:

• Artistic Description of the Project:
  https://docs.google.com/document/d/1dcvLBq6MiTKbr9Kx8Y0Cw2hZbBeBM3xjqDHFQHkZKg0/edit
• Existing Prototype:
• Duration of the Project: The project is expected to run until mid-August. The workload can be flexible and tailored to fit the availability of the developer, whether part-time or full-time.
• Budget: The budget for this specific project is approximately $15,000 USD.

Development to be Done by You:

1. You will receive a 3D Gaussian Splatting environment (.PLY) inside a Unity project using the special plugin listed above, as well as independent Gaussian Splatting scans of individual people, which should be composited in the 3D scene. Example of a capture of an individual person I did: https://lumalabs.ai/capture/eb11aae8-e24e-40a1-ba88-2b705c66569d
   Vologram dynamic volumetric capture is an alternative to also test (instead of a static Gaussian Splatting)
2. At the time being, it is not possible to rig Gaussian Splattings, but if such development would become a reality we would go for it. In the meantime, static GS should be fine and will match well visually the environment.
3. Integrate intelligent ChatBot cloned characters I would have had created inside InWorld.ai using real-world conversations (captured on camera and mic) to train the intelligent ChatBots and make a digital clone. The goal will be to match the ChatBot conversational NPCs to each scan of individual people obtaining an audio experience only (potentially text bubbles), together with a static photorealistic scan (or dynamic yet still non-interactive if we end up using Vologram). If InWorld.ai (which is based on OpenAI) poses too many challenges (training real data. French language etc.) we may have to directly use OpenAI tools and integrate them into Unity (or Unreal) in a more hardcoded manner, which could also pose some latency challenges to address.
4. Make a good UI/UX single-player experience using a smartphone as a joystick for navigation and play. Some programming to do so that it works well. Make sure the microphone works well in order to speak with the cloned NPCs. Debug, debug, debug.
5. If additional funding becomes available, we plan to explore various enhancements to the project, such as integrating narrative videos, multiplayer functionality, avatar personalization, web accessibility, virtual reality support, and distribution across multiple platforms.

Distribution as an installation (potentially a PC app):
• Equipment: a powerful PC, a smartphone for the joystick, potentially an audio headset with a microphone, and a large television or a projector.
• Provide information on running the experience (at events) for non-technical volunteers.

Some important artistic inspirations and achieving of realism:

Application Process:
If you’ve reviewed the details and your skills align, please click the following link to fill out the form. It serves as the initial screening process, so be sure to complete it thoroughly:
https://forms.gle/7YShkCARSa414C1UA

I look forward to hearing from you!

In what capacity are you hiring?
As an individual for a personal project

Where are you in the hiring process?
I’m ready to make a paid hire

Is this an independent one off job, or part of a larger project?
Single job with follow up work

For what period of time will the freelancer be needed?
More than a week

What freelancer experience level is needed? (per hour)
Mid-level (Industry Average): $35-75, Senior: $75-100, Expert: $100+