Overview

ME-Live! includes a complete set of hardware, support software, and application software. This guide will first provide you the detailed description of hardware specifications and their assembly methods; then it will introduce you how to use the system, including license installation and management, management of HoloLens device and app installation and management; lastly, it will show some built-in demo apps.

System Structure

ME-Live! is made up of two parts:

  • Live Rig: a set of movable video device with the function of getting video and location information of the real world.
  • Live WorkStation: a high performance graphics workstation used for running holographic program and synthesizing real-time holographic video.

System Structure

How It Works

ME-Live! contains two main parts, collaboration, and live streaming. The collaboration within multiple devices which is achieved by delivering messages between these devices through HoloServer makes real-time synchronization of scenes on these devices a reality. Through camera shooting and capturing real scenes, while getting the mapping of spatial position by HoloLens on the tripod, live streaming is achieved by putting them into Live Controller in Workstation to carry out real-time 3D synthesis and then output onto screens through HDMI interface.

RIG Installation

The Rig is used for obtaining the real-world images and location information and further delivering them to the workstation in order to carry out real-time synthesis.

ME-Live! Rig consists of the following main components:

  • Digital camera/video camera
  • HoloLens
  • Tripod
  • Accessories for fixing

Note: HoloLens here is responsible for providing location information. You can replace it with other devices which also can provide position information. Besides, HoloLens can be removed if you don't need real-time synchronization of location information, such as fixing the camera stand.

Rig Assembly

The purpose of the assembly is to securely secure the complete set of equipment together to minimize the effects of unstable factors such as shaking. Depending on the requirements of the working site and the choice of equipment, the ways of assembly and connection may vary. If you want to use non-recommended equipment, please make sure that you know how to connect them together. Below is a typical example of the assembly.

item Function specification
HoloLens stand Fix HoloLens firmly Aluminium stand offered by DataMesh
Digital camera Hot shoe interface for the the convenience of connecting HoloLens Sony ICLE-6500 + EPZ 16-50
Tripod/head Support camera and HoloLens firmly
Other accessories Camera can connect to HoloLens through hot shoe interface Screws for hot shoe switch

The assembly processes of Rig Suite depicted in the picture below, and the steps are as follows:

  1. Fix HoloLens on the dedicated Aluminum HoloLens Bracket.
  2. Add the Hotshoe Adapter to the HoloLens mount in Step 1 (to later in Step 5, connect the HoloLens mount to the camera).
  3. Connect the Camera to the Tripod.
  4. Fix the Hotshoe Fastener to the Camera.
  5. Finally, connect the HoloLens Mount and the Camera by connecting the Hotshoe Adapter and the Hotshoe Fastener.

RIG Assembly Diagram

Workstation Installation

Hardware Requirements

The workstation requires a certain amount of computing power, which could be satisfied by hardware sets of the given specifications, to meet the demands of ME-Live!. There would also be some restrictions regarding the hardware and software choices. Below is a list of recommended requirements for the workstation:

Item Specification Remark
CPU Intel Skylake 6700K or above OR AMD Ryzen 1700X or above
GPU NVIDIA GeForce GTX 1070 or above
  • GPU needs to support NVENC (Hardware-Accelerated Video Encoding) encoding APIs for H.264.
  • AMD GPUs currently are not supported.
Mainboard Support M.2 SSD or PCI-E SSD
Memory 16GB dual-channel DDR4 or above
Hard disk 500GB M.2 SSD or PCIE SSD
  • Recommend SAMSUNG 850 EVO 500G M.2 SSD.
Operating system Windows 10 64bit or above
  • Only 64bit system is supported.
  • Windows 7/8 and Windows Server editions are not supported.

Note 1: Visit https://developer.nvidia.com/video-encode-decode-gpu-support-matrix#Encoder to see the NVIDIA GPU support matrix.

Note 2: Single common GTX1070 card could accelerate the transcoding process up to 8x, which roughly means the system can process an 8-minutes recording in about 1 minute. The use of cards of lower GPU computation power is also possible but would result in slower video processing.

Assembly Steps

Workstation access steps are as follows:

Workstation Connection Diagram

  1. Connect the camera's HDMI Output Port to the Input Port of the capture card of the workstation, with a HDMI to Mini-HDMI cable.
  2. Connect the Micro USB port of HoloLens to one of the USB3.0 port on the workstation with a Micro USB to USB cable. (This is for the convenience of USB debugging and charging of HoloLens, and thus is optional) .
  3. Connect the workstation to the LAN Port of the Wireless Router using a Lan cable.
  4. Use an HDMI to HDMI cable to attach the Output Port of the capture card of the workstation to an external display or any other campatible screens.
  5. Let the HoloLens join the local wireless network and make sure the HoloLens and the workstation are in the same vlan.

Configurations

Network Configurations

  1. Get the Rig and Workstation ready first.
  2. Make sure a dedicated WIFI router, 802.11ac perferred, with wired connection has connected to the Rig. Router IP should be 192.168.8.1, DHCP range set to 192.168.8.100~249. Internet link is not necessary.
  3. IP address of the Workstation should be set to 192.168.8.250. We need to run both of the soft Suite and the Demo app on it.

Note: If you have other devices to collaborate, e.g. a Surface Pro as a controller, it needs to join the same WIFI network with wireless connnection, which would get an IP from pool 192.168.8.100~249 automatically. Besides, please make sure there's not a lot of WIFI devices around when you start to move the rig. WIFI latency impacts the performance.

Environment Preparation

  1. Connect everything to the WIFI router. Make sure they all have 192.168.8.x IP address, and the workstation has the fixed IP 192.168.8.250. This is very important since we fixed the network settings to ensure there's no surprise.
  2. Install the SpectatorView Test App onto your spectator view HoloLens on the Rig. You can download the SpectatorView Test App from here.

Usage Guide for MeshExpert Suite

Installation steps are as follows:

  1. Find the MeshExpert Installer.exe, double click it, and proceed according to the installation guide. If you have installed an older version, please uninstall the old version first.

  2. Restart the system or open Start MeshExpert under DataMesh entry on the Start Menu to start the service.

After the installation is complete, the DataMesh entry will be generated on the Start menu bar, which will include the service management operations, such as start and stop, and some demo programs. Click Start MeshExpert start the service, wait a moment, open the browser, enter localhost and then enter. If there is a sign-in page, the service starts successfully.

You can click Stop MeshExpert and Restart MeshExpert under the DataMesh entry on the Start menu bar to stop or restart service, respectively.

The GPU compression is preferred when performing video compression operations in the program. If the machine does not have video card or cannot use GPU for some reasons, you can use CPU for video compression by changing configuration. Instructions: go to the installation directory ("C:\Program Files\MeshExpert" by default), and then enter "HoloServer\hlstrans\configs",

Open config.toml file, find use_gpu option, and change its value to false.

License Management

Adding

Visit http://localhost/admin, and you will find the Adding License web page as follows. Select your license file and upload it to activate the features. If you don't have a valid license right now, please contact service@datamesh.com.

After the completion of the page, click on the License item on the left side of the page, you can see your license information as shown below, and also can manage your license here, such as updates and deletiona.

Managing

You can manage your license in the License page, including updates, deletions and online subscription renewals.

If your original license has expired or you have purchased a new license, you can update your license with the License Update. You can delete your current license by using the License Delete.

Device Management

Management can be used to manage HoloLens devices, application releases, licenses, and so on, which is composed of some Windows Device Portal APIs.

Adding

The Device Management module can add, delete and manage devices such as HoloLens. Specific operations are as follows:

In the case of obtaining and adding a legal license, visit http: // / admin, there will appear a page similar to the following page:

Click Add device button, a page similar to the following page appears:

On the condition that your HoloLens and Server are in the same network segment (usually connected to the same Router) and can communicate with each other, you shall fill in IP address of your HoloLens and click Next button. If it is your first time to add the device, the following page will appears with a PIN code on the HoloLens.

Enter the PIN and enter the username and password you want to create and confirm the password. Then click Pair. if pair successfully, the following page will appear with a list of devices.

beside you can conduct relevant operations on this page.

If this is not the device's first pair, enter user name and password directly.

You can click Forget your account for password reset.

Status Checking

Visit http://localhost/admin, click Device in the left to see the current status of device connected to Holo-Server. Possible status values are Online and Offline.

Removal

Click Delete to remove the device and device's occupation of license.

App Management

Release

Visit http://localhost/admin and select Application option. In the application center, you can release packaged applications to facilitate the management and installation to each HoloLens device.

Click Add Application button

Select the installation package, and click Add Dependency to add dependencies (if needed), then click Go to release the application. You can perfrom certain operations on released applications, such as delete, install, and upgrade to the specified device.

Click Go to upload, with an upload progress, and wait for the return of the status of the installation

After installation completes, uploaded apps will be shown in the Application page.

Upgrade

In the Application page, click Upgrade to upgrade the current application. The upgrade needs to pass the same name but the different version of the application. The program will check the criteria and change the version of the current application in the Application.

<

The upgrade supports the replacing the same application with the same name.

Delete

Click Delete to remove applications.

Install and uninstall

Click More to start, stop, restart, install, uninstall, upgrade sand change configuration.

  • Start

  • Stop

  • Restart

  • Install

When the user chooses an application to install, the system will change parameters of the configuration file in installation package after installation, such as \LocalAppData\ChengDu_1.2.1.0_x86__pzq3xp76mxafg\LocalState\config.ini, namely setting the field values of Server_Url, Share_Anchor_Url both to the IP of the current Server.

  • Uninstall

  • Upgrade

When the application installed in HoloLens needs to be upgraded, click Upgrade to upgrade the application and update the version number to the latest version in Application.

  • Configuration

If the user's current server IP changes, you need to click Configure to change the values Server_Url and Share_Anchor_Url accordingly.

Uninstall

Find Uninstall in DataMesh entry on Start menu and click it to uninstall it. If you cannot find Uninstall on Start menu, please go to installation directory and find Uninstall.exe, and uninstall it by double clicking the Uninstall.exe.

Note: Uninstalling MeshExpert will remove all data, including application data and licenses. If the data is important to you, be sure to make a backup.

Built-in Apps

SolarSystemExplorer

We have installed a built-in app as a sample for your reference. In this example, we implemented the most basic space anchor sharing and scene synchronization.

Below we will give you a detailed description of how to use this application.

Open the SolarSystemExplorer application, you will see the whole solar system surfaced in front of you.

Click on a specific planet, and you will be able to explore it exclusively.

You can rotate and zoom in on the entire scene at any time, just use gestures to tap and level (rotate), vertical (zoom) drag to achieve the most basic interactions. All of these operations will be synced in ME-Live! in real-time to produce a consistent scene.

You can use the voice "Open Menu" to call out the hidden menu, to adjust and share the space anchor. The operation is divided into three categories: Change Anchor, Upload Anchor and Download Anchor. Download Anchor means obtaining latest space anchor information for the Server and synchronizing with it; Upload Anchor means uploading the current HoloLens's space anchor to the Server; Change Anchor means adjusting the current scene's location and then uploading it to the Server to synchronize after the adjusting completes.

Steps of Change Anchor are as follows:

  • Through air-tapping the Change Anchor in the hidden menu, you'll enter the anchor adjustment mode, where you can see a bounding box outside the entire galaxy and a blue crystal at the reference position showing the anchor of the galaxy.

  • Eyes stare at the box, air-tap, and the object of the bounding box will change and flash indicating that it's being selected, while the top of the core crystal will appear three square buttons.

  • Watch the middle of the "Gaze" button and air-tap, you'll enter the sight movement mode, in which the HoloLens will display the surrounding environment's spatial mesh, and the entire object will move with the user's gaze and stay at the environment spatial mesh on which you gaze.

    • If there is not environmental spatial mesh in the currently-gazed position, then the object will float in the direction of eyes three meters away.
    • Air-tap again and you will exit the sight movement mode. Then re-choose the status of the object.
  • Gaze at Move button in the left and air-tap, and you will enter the manual movement mode and the crystal will appear on the mobile logo.

    Note: When using HoloLens, use the "press the finger and move" to manipulate the object, you can adjust the position of the object in three axes.

  • Gaze at Rotate button and air-tap, and you'll enter the manual rotation mode and then the crystal appears back to the rotation mark.

    Note: When using HoloLens, use the "press the finger and move" to manipulate the object and you can rotate object in three axes.

After completing the adjustment of the space anchor, air-tapping at the blank area outside of the bounding box will exit the anchor edit mode, the outer circumference and the crystal will disappear, and the location of the galaxy will be stored. When you open the program in HoloLens next time, the galaxy will return to this position.