Feature Request: SMPL format export

[jaykup1]

Exported .npy has SMPL parameters but it would be great if it also get exported as pkl file with SMPL parameters.

[GuyTevet]

Hi @jaykup1 ,

You can replace

motion-diffusion-model/visualize/vis_utils.py

Line 66 in 14d0f79

np.save(save_path, data_dict)

With pickle.dump(*) if that was your intention.

[jjp9624022]

this is blender addon to opt smpl model here https://github.com/Meshcapade/SMPL_blender_addon
The last version allows import animations in .npz format
how to trans？It would be great if it could work

[jaykup1]

Hi @jaykup1 ,

You can replace

motion-diffusion-model/visualize/vis_utils.py

Line 66 in 14d0f79

np.save(save_path, data_dict)

With pickle.dump(*) if that was your intention.

Not exactly: https://meshcapade.wiki/SMPL

Anyway to export with this skeleton layout? (so we can use https://github.com/softcat477/SMPL-to-FBX to convert that to FBX for game engines)

[TREE-Ind]

Definitely going to try out the SMPL to FBX converter later tonight. As a workaround we've had great success simply running the rendered mesh sequence through AI mocap solutions to get game ready rigged animations, Plask, Deepmotion, etc.

[jjp9624022]

Hi @jaykup1 ,
You can replace

motion-diffusion-model/visualize/vis_utils.py

Line 66 in 14d0f79

np.save(save_path, data_dict)

With pickle.dump(*) if that was your intention.

Not exactly: https://meshcapade.wiki/SMPL

Anyway to export with this skeleton layout? (so we can use https://github.com/softcat477/SMPL-to-FBX to convert that to FBX for game engines)

not work! different shape
results.npy:motion (3, 22, 3, 120)
SMPL-to-FBX sample.pkl motion (127, 72)

[jaykup1]

not work! different shape results.npy:motion (3, 22, 3, 120) SMPL-to-FBX sample.pkl motion (127, 72)

The export file has to be rearranged to SMPL skeleton layout before fbx conversion, help would be greatly appreciated.

[GuyTevet]

I'll try to clarify, please let me know if it helps:

results.npy - only includes joint positions - this is not what you are looking for.

after running visualize.render_mesh you will get sample##_rep##_smpl_params.npy with smpl parameters. In details:

            'motion': [25, 6, frames], - this is both SMPL, thetas, and root translation, you can ignore it.
            'thetas': [24, 6, frames] - SMPL thetas represented in 6d rotations
            'root_translation': [3, frames] - Root translation
            'faces':  - SMPL faces list
            'vertices': - SMPL vertices locations per frame
            'text': - text prompt
            'length': - number of frames

• I think 72 stands for 24 joint * 3 (axis angle representation)
• The fields relevant for you are thetas and root_translation you can ignore the rest.
• Alternatively, instead of keyframing SMLP joints you can directly keyframe the mesh's vertices locations, they are also provided in this .npy file

Can you detail here the data structure that the SMPL addon is expecting, and we will try to code an adapter between the two formats (and of course welcome you to send a pull request if you figure it out yourself).

Hope this helps,

[jaykup1]

Thank you for the reply,

• Yes 72 stands for 24 joint * 3 dimension for the rotation vector.
• The pkl file contains a dictionary with: smpl_poses and smpl_trans:
• 'smpl_trans': [frames, 3] - Root translation
• 'smpl_poses' : [frames, 72] - thetas
  • Joint order expected is:
    0:Pelves 1:L_hip 2:R_hip 3:Spine1 4:L_Knee 5:R_Knee 6:Spine2 7:L_Ankle 8:R_Ankle 9:Spine3 10:L_Foot 11:R_Foot 12:Neck 13:L_Collar 14:R_Collar 15:Head 16:L_Shoulder 17:R_Shoulder 18:L_Elbow 19:R_Elbow 20:L_Wrist 21:R_Wrist 22:L_Hand 23:R_Hand

[GuyTevet]

Cool! If that's the case, the thetas and root_translation have all the information you need. You will just need to convert 6d into axis angles. One way to do so is to concatenate those two functions in our repo:

motion-diffusion-model/utils/rotation_conversions.py

Lines 513 to 534 in 14d0f79

	def rotation_6d_to_matrix(d6: torch.Tensor) -> torch.Tensor:
	"""
	Converts 6D rotation representation by Zhou et al. [1] to rotation matrix
	using Gram--Schmidt orthogonalisation per Section B of [1].
	Args:
	d6: 6D rotation representation, of size (*, 6)
	Returns:
	batch of rotation matrices of size (*, 3, 3)
	[1] Zhou, Y., Barnes, C., Lu, J., Yang, J., & Li, H.
	On the Continuity of Rotation Representations in Neural Networks.
	IEEE Conference on Computer Vision and Pattern Recognition, 2019.
	Retrieved from http://arxiv.org/abs/1812.07035
	"""
	a1, a2 = d6[..., :3], d6[..., 3:]
	b1 = F.normalize(a1, dim=-1)
	b2 = a2 - (b1 * a2).sum(-1, keepdim=True) * b1
	b2 = F.normalize(b2, dim=-1)
	b3 = torch.cross(b1, b2, dim=-1)
	return torch.stack((b1, b2, b3), dim=-2)

motion-diffusion-model/utils/rotation_conversions.py

Lines 434 to 447 in 14d0f79

	def matrix_to_axis_angle(matrix):
	"""
	Convert rotations given as rotation matrices to axis/angle.
	Args:
	matrix: Rotation matrices as tensor of shape (..., 3, 3).
	Returns:
	Rotations given as a vector in axis angle form, as a tensor
	of shape (..., 3), where the magnitude is the angle
	turned anticlockwise in radians around the vector's
	direction.
	"""
	return quaternion_to_axis_angle(matrix_to_quaternion(matrix))

[Tony2371]

So, will there be a script that converts sample##_rep##_smpl_params.npy to FBX or something like that?

BTW, awesome project, guys! Thanks a lot for your efforts.

[amb]

results.npy - only includes joint positions - this is not what you are looking for.

after running visualize.render_mesh you will get sample##_rep##_smpl_params.npy with smpl parameters. In details:

Does this process add new information? I've been building armatures and animations just from the results.npy manually with some quaternion magic. Would sample##_rep##_smpl_params.npy yield better results?

[TREE-Ind]

@amb very cool! Are you able to share any insight or code snippet as to how you accomplished this? We're getting close but it would help us greatly, no worries either way thx.

[amb]

@TREE-Ind Sure, it's just a proof of concept Blender script tho. Here you go: https://gist.github.com/amb/69cc8396bc61cc59a7e819aed6b21f34
The script has "import bpy" so technically it's also GPL3 license.

[TREE-Ind]

@amb Wow nice work :D

[jjp9624022]

@TREE-Ind Sure, it's just a proof of concept Blender script tho. Here you go: https://gist.github.com/amb/69cc8396bc61cc59a7e819aed6b21f34 The script has "import bpy" so technically it's also GPL3 license.

nice work! but....... Like you said ，# TODO: Is this really the best way to create armature in Blender?
you can trans the pose from the initial posture of the human skeleton someting like rigfly

[jaykup1]

Work in progess...:

@GuyTevet How can I get the 6d out of 'thetas': self.motions['motion'][0, :-1, :, :self.real_num_frames], for rotation_6d_to_matrix ?

[GuyTevet]

@jaykup1 those are the 6d rotations, but in SMPL convention.
I can make a guess why this is not working -
(1) it can be an issue with the conversion
(2) SMPL angles are relative to SMPL rest pose (T-pose I think), maybe the rest pose of your character is different. In this case, I can suggest, you put the SMPL character (using SMPL addon) side by side with your character, and align between them.

You said rotation_6d_to_matrix doesn't work well, what exactly is the issue there?

[GuyTevet]

@jaykup1 There is knowledge regarding how to convert SMPL angles to other characters (not in my brain, unfortunately). I suggest contacting the SMPL team or watching their tutorials on youtube. If you manage to resolve this, I will be very happy to know about it!

[jaykup1]

@jaykup1 those are the 6d rotations, but in SMPL convention. I can make a guess why this is not working - (1) it can be an issue with the conversion (2) SMPL angles are relative to SMPL rest pose (T-pose I think), maybe the rest pose of your character is different. In this case, I can suggest, you put the SMPL character (using SMPL addon) side by side with your character, and align between them.

You said rotation_6d_to_matrix doesn't work well, what exactly is the issue there?

@GuyTevet I only get TypeErrors, AttributeErrors (because I don't know what I'm doing), I just reshaped it (reshape(self.real_num_frames, 72), (reshape(self.real_num_frames, 3) to see if it's working.

[GuyTevet]

The shape of thetas is [24, 6, frames]. 6 stands for the 6d, you can look in our code or in ACTOR or MotionCLIP repos for references on how to use those functions.

[mmdrahmani]

Hi
Thank you for the excellent work. I have tried the code and it works just fine. I even used the mesh object sequence in blender and that was also fine (really interesting results).
I only had a basic question, regarding how to use the npy file in the blender SMPL or SMPL-X add on in order to create animations (similar to animations for the paper). I think this is mostly my fault because I know little about blender. I appreciate any information, or link to blender tutorials. I watched several SMPL/X blender tutorials on youtube but none of them addressed my problem.
All the bests
Mohammad

[mmdrahmani]

lso GPL3 license

Thanks for the code. This is amazing!
How should I used SMPL neutral body instead of armature?
Thanks

[alextitonis]

@amb
Tried to use the script and exported the fbx, but the result was an orb with spykes around moving
Is there a different way to export it?
bpy.ops.export_scene.fbx(filepath="./tata.fbx", use_selection=True, add_leaf_bones=False)

[jaykup1]

Conversion of data to .pkl file for 'softcat477/SMPL-to-FBX/'

@jjp9624022 @TREE-Ind @Tony2371 @amb @mmdrahmani @alextitonis tell me if there are any problems, thanks

[alextitonis]

It works like charm awesome @jaykup1

[TREE-Ind]

Getting the FBX python SDK setup correctly was a pain but can also confirm the conversion works for us as well.

[mmdrahmani]

Conversion of data to .pkl file for 'softcat477/SMPL-to-FBX/'

@jjp9624022 @TREE-Ind @Tony2371 @amb @mmdrahmani @alextitonis tell me if there are any problems, thanks

This is cool! Thank you.
I ran the code and converted and saved the data into a pickle file. But I am still not sure how to use this pickle file in Blender. I have SMPL and SMPL-X addon installed on blender, but I don't see how to import animations saved in the pickle file.
Thank you for your support.

[jaykup1]

@mmdrahmani use the pkl file with this https://github.com/softcat477/SMPL-to-Fbx

Change 'FbxTime.eFrames60's in SMPL-to-FBX/FbxReadWriter.py to 'FbxTime.eFrames30'

Known issues: Glitches below 30 fps exports in 'softcat477/SMPL-to-FBX/' (20 fps expected)

[TREE-Ind]

mdm2ue5.mp4

Importing motion diffusion animations into UE5 at runtime now thx to the conversion code :D

[jaykup1]

Importing motion diffusion animations into UE5 at runtime now thx to the conversion code :D

you can half the time of export by reducing 'self.num_smplify_iters = 150' in simplify_loc2rot.py to 50. (30 seemed fine), I will probably make a simple gradio gui and push it.

[ivansabolic]

The conversion code works awesome, thanks!

Any suggestion what would be the best approach to bring down the file size of generated .fbx? Quality loss is acceptable till some point, I am just not sure if I should be playing with SMPL parameters or something else.

[jaykup1]

The conversion code works awesome, thanks!

Any suggestion what would be the best approach to bring down the file size of generated .fbx? Quality loss is acceptable till some point, I am just not sure if I should be playing with SMPL parameters or something else.

Not possible (that I know of), reducing the imported fbx size breaks the animation calculation. If you are using a game engine you can export the animation. In unity engine, animation is ~4 MB (0.3 MB if you zip it).

[TREE-Ind]

@babotrojka Not sure if this will work for your setup but we had good success converting the fbx to gltf while retaining the animation data which reduces the file size substantially.

[mmdrahmani]

Thank again.
I tried to work with this https://github.com/softcat477/SMPL-to-Fbx as you suggested. But setting up python fbx is quite complicated. I have not been able to configure it on my macbook. The FBX-python instructions are very cryptic.
I followed the instructions and installed fbx sdk on my macbook and then the smpl-to-fbx requirements in a new conda environment. Then while running the Python Convert.py I got this error (without more info):

['gBR', 'gPO', 'gLO', 'gMH', 'gLH', 'gHO', 'gWA', 'gKR', 'gJS', 'gJB']
['sBM', 'sFM']
['0', '1', '2', '3', '4', '5']
Segmentation fault: 11

Do you have any ideas why I am getting this error? My guess the FBX sdk configuration was not done properly, although I followed the instructions provided.
Thanks

[yotaro-shimose]

Thank again. I tried to work with this https://github.com/softcat477/SMPL-to-Fbx as you suggested. But setting up python fbx is quite complicated. I have not been able to configure it on my macbook. The FBX-python instructions are very cryptic. I followed the instructions and installed fbx sdk on my macbook and then the smpl-to-fbx requirements in a new conda environment. Then while running the Python Convert.py I got this error (without more info):

['gBR', 'gPO', 'gLO', 'gMH', 'gLH', 'gHO', 'gWA', 'gKR', 'gJS', 'gJB'] ['sBM', 'sFM'] ['0', '1', '2', '3', '4', '5'] Segmentation fault: 11

Do you have any ideas why I am getting this error? My guess the FBX sdk configuration was not done properly, although I followed the instructions provided. Thanks

The first three lines are the output of these lines

For Segmentation fault: 11, I'm not confident. But if not tried, make sure you installed the correct version of "Python".

[yh675]

@mmdrahmani use the pkl file with this https://github.com/softcat477/SMPL-to-Fbx

Change 'FbxTime.eFrames60's in SMPL-to-FBX/FbxReadWriter.py to 'FbxTime.eFrames30'

Known issues: Glitches below 30 fps exports in 'softcat477/SMPL-to-FBX/' (20 fps expected)

@jaykup1
In line 84 of the conversion script

smpl_trans = np.array(trans_raw).reshape(self.real_num_frames, 3)*np.array([100, 1, 100])

How come the y coordinate is only scaled by 1 and not 100?

[yh675]

I am loading animations into Unreal Engine using the conversion to pkl and SMPL-to-FBX repo.

However, I would like to control the relative location of the foot joints to the ground floor in the simulator.

I tried to use the coordinates given in results.npy for one animation, according to which the left foot is at a height of 0.388 cm in the first frame (I multiplied the scalars in results.npy by 100).

However, when I load the animation into Unreal it looks like this:

Where the feet are clearly 8-9 cm above the animations origin (m_avg_root) in it's local coordinate frame.

How can I get the accurate local xyz coordinates of the joints in the animations? Did anyone else have a similar issue?

[yh675]

I resolved the issue by mapping the translation of m_avg_root to m_avg_Pelvis.

Fixed by changing motion-diffusion-model.visualize.vis_utils from:
smpl_trans = np.array(trans_raw).reshape(self.real_num_frames, 3)*np.array([100, 1, 100]) ->
smpl_trans = np.array(trans_raw).reshape(self.real_num_frames, 3)*np.array([100, 100, 100])

And by changing the line in SMPL-to-FBX/FbxReadWriter.py from:
name = "m_avg_root" -> name = "m_avg_Pelvis"

[tommyshelby4]

I made the change you suggested but still see an offset of 20cm in the vertical axis, the foot-ground contact does not exist

[yh675]

I made the change you suggested but still see an offset of 20cm in the vertical axis, the foot-ground contact does not exist

You also have to take into account the lowest joint location in results.npy which may not be at 0. Then load the mesh into Unreal Engine with the lowest joint location offset from the floor height.

[LEEYEONSU]

mdm2ue5.mp4
Importing motion diffusion animations into UE5 at runtime now thx to the conversion code :D

hello. i was wondering the way you import motion-diffusion-model output to Unreal Engine.

I am now struggling with retargeting to UE5.0 .

If you have a code could you share it or explain it.

Thank you for reading.

[fwwucn]

I made the change you suggested but still see an offset of 20cm in the vertical axis, the foot-ground contact does not exist

You also have to take into account the lowest joint location in results.npy which may not be at 0. Then load the mesh into Unreal Engine with the lowest joint location offset from the floor height.

I have the same issue, but I don't think I should always adjust the lowest joint location to 0, because some frames may have jumping action (two feet are away from ground).
Is there any flag for each frame which can indicate whether the lowest joint location of current frame should be on the ground plane or not?

[tshrjn]

Thank again. I tried to work with this https://github.com/softcat477/SMPL-to-Fbx as you suggested. But setting up python fbx is quite complicated. I have not been able to configure it on my macbook. The FBX-python instructions are very cryptic. I followed the instructions and installed fbx sdk on my macbook and then the smpl-to-fbx requirements in a new conda environment. Then while running the Python Convert.py I got this error (without more info):
['gBR', 'gPO', 'gLO', 'gMH', 'gLH', 'gHO', 'gWA', 'gKR', 'gJS', 'gJB'] ['sBM', 'sFM'] ['0', '1', '2', '3', '4', '5'] Segmentation fault: 11
Do you have any ideas why I am getting this error? My guess the FBX sdk configuration was not done properly, although I followed the instructions provided. Thanks

The first three lines are the output of these lines

For Segmentation fault: 11, I'm not confident. But if not tried, make sure you installed the correct version of "Python".

I seem to be having the same issue,

[tshrjn]

@mmdrahmani @yotaro-shimose I was able to resolve Segfaults as mentioned in this PR.

[tshrjn]

@TREE-Ind You mentioned converting fbx to gltf - could you share how you did this? And also, how can we transfer the animation to other meshes?

@babotrojka Not sure if this will work for your setup but we had good success converting the fbx to gltf while retaining the animation data which reduces the file size substantially.

[tshrjn]

@mmdrahmani @yh675 @jaykup1 @GuyTevet After we get the FBX file, how are you retargetting that motion to other assets like the one shown in the video?

[kruzel]

hi,
I'm looking for a way to create animation without going thru visualize.render_mesh.

does anyone know how to get the root pose and rotations from the model output here in generate.py:

[dj-kefir-siorbacz]

@TREE-Ind Sure, it's just a proof of concept Blender script tho. Here you go: https://gist.github.com/amb/69cc8396bc61cc59a7e819aed6b21f34 The script has "import bpy" so technically it's also GPL3 license.

If I see correctly, this script creates a Blender armature based on positional data from .npy. Then, for every frame it aligns the pose bones of this armature to match the positions of .npy joints.

There are three issues with this method:

• the twists (rotations along the axis of the bone) of rest pose (edit mode) bones are not specified explictly.
• the twists of pose (pose mode) bones are not specified explictly.
• the rest pose positions of bones are the same as the positions of pose in the first frame.

TL;DR this method is worthless for retargetting/driving other 3D models; it fakes the rotational information.
You are better off treating the positional data of the original results.npy as the end-effector bones of inverse kinematics.

@amb Tried to use the script and exported the fbx, but the result was an orb with spykes around moving Is there a different way to export it? bpy.ops.export_scene.fbx(filepath="./tata.fbx", use_selection=True, add_leaf_bones=False)

Blender's FBX export is pretty bad. I wouldn't rely on it. You are better off exporting to .bvh using Blender, then importing this .bvh to some software that can both import .bvh and export .fbx well (for example, Autodesk software, e.g. Motion Builder). That way you are sure your .fbx will work in other software.
If you're fine with .gltf or .glb, then Blender does a fine job exporting them.

[ciroanni]

Hi, I am trying to convert MDM model output to fbx files so that I can import them to Unity. I have tried several methods, starting from the npy file I used the smpl2bvh script to get the mocap and then from Blender export it to FBX but I get strange effects as shown here

Registrazione.2024-10-23.172800.mp4

I assume it is due to the fact that the npy file only contains the positions of the joints frame by frame and not the rotations so when I have more complicated animations where the character has to rotate, turn, etc... it just tries to get the joints to that position. So I tried the repo that you have previously linked i.e. SMPL-TO-FBX. This definitely works better but it is very slow, it takes me at least 3 minutes and also I still have small glitches as you can see here.

Registrazione.2024-10-25.095501.mp4

So when I have an easy input like “Walk forward”, paradoxically it comes better with the first method, in fact I don't have those little head movements for example and it is much faster. Could you tell me how come I have those problems with the second method? Also, do you have a way to implement the first method by also calculating the rotations but avoiding the SMPLify algorithm which takes a lot of time? Thank you very much in advance